mygrad.linalg.norm

mygrad.linalg.norm(x: ArrayLike, ord: Optional[Union[int, float]] = None, axis: Optional[Union[int, Tuple[int]]] = None, keepdims: bool = False, *, nan_to_num: bool = True, constant: Optional[bool] = None) → Tensor

Vector norm.

This function is an infinite number of vector norms (described below), depending on the value of the ord parameter.

In contrast to numpy.linalg.norm, matrix norms are not supported.

This docstring was adapted from that of numpy.linalg.norm [1].

Parameters

xArrayLike

Input tensor. If axis is None, then x must be 1-D unless ord is None. If both axis and ord are None, the 2-norm of x.ravel will be returned.

ordOptional[Union[int, float]]

Order of the norm (see table under Notes). inf means numpy’s inf object. The default is None.

axisOptional[Union[int, Tuple[int]]]

If axis is an integer, it specifies the axis of x along which to compute the vector norms. The default is None.

keepdimsbool, optional (default=False)

If this is set to True, the axes which are normed over are left in the result as dimensions with size one. With this option the result will broadcast correctly against the original x.

nan_to_numbool, optional (default=True)

If True then gradients that would store nans due to the presence of zeros in x will instead store zeros in those places.

constantOptional[bool]

If True, this tensor is treated as a constant, and thus does not facilitate back propagation (i.e. constant.grad will always return None).

Defaults to False for float-type data. Defaults to True for integer-type data.

Integer-type tensors must be constant.

Returns

Tensor

Norm(s) of the vector(s).

Notes

For values of ord < 1, the result is, strictly speaking, not a mathematical ‘norm’, but it may still be useful for various numerical purposes.

The following norms can be calculated:

ord	norm for vectors
inf	max(abs(x))
-inf	min(abs(x))
0	sum(x != 0)
1	as below
-1	as below
2	as below
-2	as below
other	sum(abs(x)**ord)**(1./ord)

The Frobenius norm is given by [1]:

\(||A||_F = [\sum_{i,j} abs(a_{i,j})^2]^{1/2}\)

The nuclear norm is the sum of the singular values.

Both the Frobenius and nuclear norm orders are only defined for matrices and raise a ValueError when x.ndim != 2.

References

1(1,2)

Retrieved from: https://numpy.org/doc/stable/reference/generated/numpy.linalg.norm.html

2

G. H. Golub and C. F. Van Loan, Matrix Computations, Baltimore, MD, Johns Hopkins University Press, 1985, pg. 15

Examples

>>> import mygrad as mg
>>> x = mg.tensor([[1.0, 2.0, 3.0],
...                [1.0, 0.0, 0.0]])
>>> l2_norms = mg.linalg.norm(x, axis=1, ord=2)
>>> l2_norms
Tensor([3.74165739, 1.        ])

The presence of the elementwise absolute values in the norm operation means that zero-valued entries in any of input vectors have an undefined derivative. When nan_to_num=False is specified these derivatives will be reported as nan, otherwise they will be made to be 0.0.

>>> l2_norms = mg.linalg.norm(x, axis=1, ord=2, nan_to_num=True)
>>> l2_norms.backward()
>>> x.grad
array([[0.26726124, 0.53452248, 0.80178373],
       [1.        ,        nan,        nan]])

This is rigorously true, but is often not the desired behavior in autodiff applications. Rather, it can be preferable to use 0.0 to fill these undefined derivatives. This is the default behavior, when nan_to_num is not specified.

>>> l2_norms = mg.linalg.norm(x, axis=1, ord=2, nan_to_num=False)  # default setting: `nan_to_num=False`
>>> l2_norms.backward()
>>> x.grad
array([[0.26726124, 0.53452248, 0.80178373],
      [1.        ,          0.,         0.]])

L1 norms along each of the three columns:

>>> mg.linalg.norm(x, axis=0, ord=1)
Tensor([2., 2., 3.])