Vectors norms are a very important concept in machine learning. In a slightly unorthodox way, we will show the intuition behind this concept. In order to do this we will generate a large number of random vectors in two dimensions known as R2.
The following Python functions creates random vectors and plots them:
Not all the code for plots will be displayed on this document in the interest of space, find them here if you want to recreate them.
import matplotlib.pyplot as plt
import numpy as np
plt.style.use('seaborn-whitegrid')def random_vector2d(n):
x = np.linspace(-10,10,n)+np.random.uniform(-10,10,size=n)
y = np.linspace(-10,10,n)+np.random.uniform(-10,10,size=n)
vecs = [np.array([i,j]) for i,j in zip(x,y)]
return vecsdef plot_vector2d(vector2d, ax,color=np.random.random(3),**options):
return ax.arrow(0, 0, vector2d[0], vector2d[1],
head_width=0.3, head_length=0.3,
length_includes_head=True,color=color,**options)
Create and display 1000 random vectors
n= 1000
vecs = random_vector2d(n)fig,ax =plt.subplots(figsize=(13,8))
for vec in vecs:
plot_vector2d(vec,ax=ax,color=np.random.random(3),linewidth=4)
plt.axis([-20,20, -20, 20])
plt.axis('off')
plt.title('Random Vectors in $\mathbb{R}^2$',fontsize=20)
So although we have constrained the length of these vectors, notice that they can hit any point in R2 if we were to multiply them by a scalar. As the title suggests, we are going to normalize these vectors and inspect the resulting shape.
L2 norm.
The L2 norm is simply the length/magnitude of a vector. The formula for calculating it is given below:
Recalling the formula for the length of a 2d vector:
notice the formulae above return exactly the same answer since:
In Python we can calculate this norm using the following function:
def l2_norm(vector):
return (np.sum([i**2 for i in vector]))**(1/2)L2 Normalized Vectors
A normalized vector will have a length of 1 and is often referred to as the unit vector. The formula to normalize a vector is given below:
Example
Take the following vector v⃗
Calculate The L2 norm
Calculate the L2 Normalized Vector
Check the length of the resulting vector is 1
What does that look like?
The word norm is derived from the Latin normalis “in conformity with rule”. In order to discover the rule behind this normalization process we will normalize the random vectors we created at the beginning on the document and plot them.
Clearly a pattern emerges when we calculate the L2 normalized vectors as shown below:
What rules can we derive from this?
1) When we normalize a vector v⃗ the normalized vector vˆ will have a length of 1.
2) The Normalized vector will have the same direction as the original vector.
3) When thinking of L2 normalization in 2 dimensions, we should think: unit circle
L1 Norm
The L1 norm also known as the taxi-cab norm or the Manhattan norm. Is another common way to normalize vectors. The formula is given below:
It may be slightly perplexing why the two exponents of 1 have been included in the formula above. However, it will soon become clear. Note that we could just as easily have written the formula above as:
Python function for L1 norm
def l1_norm(vec):
return np.sum([np.abs(i) for i in vec])Example
Take the following vector v⃗
Take the L1 norm
Normalize the vector by the L1 norm
Which looks like:
Let’s normalize the random vectors we created at the beginning of the document using the L1 norm and inspect the resulting shape.
As with the L2 norm we see an nice shape emerge after normalizing the vectors. Whereas the L2 norm vectors produced a circle, the L1 shows a diamond.
What can we say about the L1 norm?
1) The L1 norm will always be greater than or equal to the L2 norm. Why? Think back to the fact that the L1 norm is also known as the taxicab norm, we could think of the L2 norm as the as the crow flies distance, whereas the L1 norm is equivalent to taking a taxi along streets that requires turns.
2) When we normalize a 2d vector by the L1 norm. The resulting vector’s elements |x|+|y|=1
3) When thinking of the L1 norm in 2d we should think of a diamond
In this section we will compare the formulae for the L1 and L2 norm. Recall the formula for the L1 norm for which a seemingly unnecessary exponents were included. Observe the formulae for the L1 and L2 norm below:
You may notice a pattern. Although Let’s add one more to show this pattern more clearly:
The P-Norm
Hopefully the pattern has become clear, we simply take each element of the vector to the power p. We can show a general formula for the norms we have so far:
Infinity Norm
Clearly there are too many norms for us to show the all shapes we would get from normalizing by the all the positive integers. However, we can still get an idea for what they may look like. Consider what would happen if we increased p towards infinity
The resulting norm is known as the infinity norm. It may not be clear how exactly we got from the formula for the p-norm to just taking absolute value from a vector. To understand why this is true, consider taking the elements of the a 2d vector and consider taking said elements to the power of p, as p gets very large the larger of the two elements will begin to dominate the calculation.
Examples
Take the following vectors and find the infinity norms for each.
So clearly the infinity norm is the easiest to calculate of all the norms we have considered so far, although it does require more thinking to get a grasp of it!
To normalize a vector by the infinity norm, we would simply divide the vector elements by the maximum absolute value. Lets take v⃗ from above as an example:
Infinity norm in Python
def infinity_norm(vec):
return np.max([abs(i) for i in vec])Let’s normalize and plot the random vectors we have been working with so far.
If there is nothing else you take away from this article, just remember the following:
L1 norm: Diamond
L2 norm: Cirlce
Infinity norm: Square
Having an understanding of this will make many types of machine learning models easier to understand.
See all articles on www.codearmo.com and feel free to comment if you notice anything mistakes!
