Norm and scalar product

 A vector space is a set with addition and scalar products. We can introduce more structures into this set. Here, we define a vector's norm (length) and the scalar product (dot product) between vectors.

Definition (Norm, length, modulus)

The norm (also called length or modulus) of the vector $\mathbf{x}=(x_1,\cdots ,x_n)$ is the real number defined by

$$\begin{array}{}\text{(eq:norm)}& \Vert \mathbf{x}\Vert =\sqrt{\sum _{i=1}^n{x}_i^2}=\sqrt{x_1^2+x_2^2+\cdots +x_n^2}.\end{array}$$

Remark. The norm of a vector, as defined above, is also known as the 2-norm or the Euclidean norm of the vector. More generally, for $k\in \mathbb{N}$, the $k$-norm of the vector $\mathbf{x}\in {\mathbb{R}}^n$ is defined as

$$\Vert \mathbf{x}{\Vert }_k=\sqrt[k]{\sum _{i=1}^n|x_i{|}^k}.$$

□

Example. When $n=1$, the norm of $\mathbf{x}=(x_1)$ is given by

$$\Vert \mathbf{x}\Vert =\sqrt{x_1^2}=|x_1|$$

which coincides with the absolute value of the real number $x_1$.

When $n=2$, the norm of of $\mathbf{x}=(x_1,x_2)$ is

$$\Vert \mathbf{x}\Vert =\sqrt{x_1^2+x_2^2}$$

which is the distance between the origin and $\mathbf{x}=(x_1,x_2)$ in the 2-dimensional space. 

We can consider the modulus of a vector defined by (eq:norm) as a generalization of the distance between the origin and $\mathbf{x}$ in the $n$-dimensional space. □

Definition (Unit vector)

Let $\mathbf{x}\in {\mathbb{R}}^n$. $\mathbf{x}$ is called a unit vector if $\Vert \mathbf{x}\Vert =1$. 

We can define a ``multiplication'' between two vectors to obtain a scalar.

Definition (Scalar product)

Let $\mathbf{x}=(x_1,\cdots ,x_n)$, $\mathbf{y}=(y_1,\cdots ,y_n)\in {\mathbb{R}}^n$. The scalar product of these two vectors is defined by

$$⟨\mathbf{x},\mathbf{y}⟩=\sum _{i=1}^n{x}_i{y}_i=x_1{y}_1+x_2{y}_2+\cdots +x_n{y}_n.$$

Remark. The scalar product is also called the inner product or dot product, and they can be denoted as $\mathbf{x}\cdot \mathbf{y}$ or $(\mathbf{x},\mathbf{y})$ or $⟨\mathbf{x}|\mathbf{y}⟩$. □

Remark. The scalar product may be considered as a map

$$⟨\cdot ,\cdot ⟩:{\mathbb{R}}^n\times {\mathbb{R}}^n\to \mathbb{R}$$

that maps a pair of vectors to a real number. □

You should prove the following lemma.

Lemma

Let $\mathbf{x},\mathbf{y},\mathbf{z}\in {\mathbb{R}}^n$ and $\lambda \in \mathbb{R}$.

1. $\Vert \mathbf{x}{\Vert }^2=⟨\mathbf{x},\mathbf{x}⟩$.
2. $\Vert \lambda \mathbf{x}\Vert =|\lambda |\cdot \Vert \mathbf{x}\Vert$.
3. $⟨\mathbf{x},\mathbf{y}⟩=⟨\mathbf{y},\mathbf{x}⟩$.
4. $⟨\mathbf{x}+\mathbf{y},\mathbf{z}⟩=⟨\mathbf{x},\mathbf{z}⟩+⟨\mathbf{y},\mathbf{z}⟩$.
5. $⟨\mathbf{x},\mathbf{y}+\mathbf{z}⟩=⟨\mathbf{x},\mathbf{y}⟩+⟨\mathbf{x},\mathbf{z}⟩$.
6. $\lambda ⟨\mathbf{x},\mathbf{y}⟩=⟨\lambda \mathbf{x},\mathbf{y}⟩$.
7. $\lambda ⟨\mathbf{x},\mathbf{y}⟩=⟨\mathbf{x},\lambda \mathbf{y}⟩$.

The next theorem provides a geometric interpretation of the scalar product.

Theorem (Angle between vectors in ${\mathbb{R}}^2$)

Suppose that $\mathbf{u},\mathbf{v}\in {\mathbb{R}}^2$, then 

$$⟨\mathbf{u},\mathbf{v}⟩=\Vert \mathbf{u}\Vert \cdot \Vert \mathbf{v}\Vert \cdot \cos \theta$$

where $\theta$ is the angle between the vectors $\mathbf{u}$ and $\mathbf{v}$.

Proof. First, consider the case where $\mathbf{u}$ and $\mathbf{v}$ are unit vectors, namely, $\Vert \mathbf{u}\Vert =\Vert \mathbf{v}\Vert =1$.

Then we can write $\mathbf{u}=(u_1,u_2)=(\cos \varphi ,\sin \varphi )$ and $\mathbf{v}=(v_1,v_2)=(\cos \psi ,\sin \psi )$ where $\varphi$ and $\psi$ are angles of $\mathbf{u}$ and $\mathbf{v}$ from $(1,0)$, respectively. Then

$$⟨\mathbf{u},\mathbf{v}⟩=\cos \varphi \cos \psi +\sin \varphi \sin \psi =\cos (\varphi -\psi ).$$

Note that $\varphi -\psi$ is the angle between $\mathbf{u}$ and $\mathbf{v}$ so that $\theta =\varphi -\psi$, and the claim holds.

Next, consider the general case where $\mathbf{u}$ and $\mathbf{v}$ are not unit vectors. We can write $\mathbf{u}=\Vert \mathbf{u}\Vert \hat{\mathbf{u}}$ and $\mathbf{v}=\Vert \mathbf{v}\Vert \hat{\mathbf{v}}$ where $\hat{\mathbf{u}}$ and $\hat{\mathbf{v}}$ are unit vectors. Thus,

$$⟨\mathbf{u},\mathbf{v}⟩=\Vert \mathbf{u}\Vert \cdot \Vert \mathbf{v}\Vert ⟨\hat{\mathbf{u}},\hat{\mathbf{v}}⟩=\Vert \mathbf{u}\Vert \cdot \Vert \mathbf{v}\Vert \cos \theta$$

as required. ■

Remark. Let's look at this theorem from a different perspective. Recall that we constructed $\mathbb{C}$ from ${\mathbb{R}}^2$. 

See also: Constructing complex numbers.

So, a complex number is actually a vector in ${\mathbb{R}}^2$. The unit vector $\mathbf{u}$ corresponds to the complex number $e^{i\varphi }$, and the unit vector $\mathbf{v}$ corresponds to $e^{i\psi }$.

Note that the scalar product

$$⟨\mathbf{u},\mathbf{v}⟩=⟨(u_1,u_2),(v_1,v_2)⟩=u_1{v}_1+u_2{v}_2$$

is the real part of 

$$(u_1-i{u}_2)(v_1+i{v}_2)=(u_1{v}_1+u_2{v}_2)+i(u_1{v}_2-u_2{v}_1),$$

which, in turn, is the real part of

$$e^{-i\varphi }{e}^{i\psi }=e^{-i(\varphi -\psi )}=\cos (\varphi -\psi )-i\sin (\varphi -\psi ).$$

□

We can prove the same theorem for ${\mathbb{R}}^3$ (You should try to prove it). We want the same result for any dimension, ${\mathbb{R}}^n$. How do we prove that? No, we don't prove that. Instead, we define the angle between two vectors in terms of the scalar product. But before that, we need the following theorem.

Theorem (Cauchy-Schwarz inequality)

Let $\mathbf{x},\mathbf{y}\in {\mathbb{R}}^n$. Then

$$|⟨\mathbf{x},\mathbf{y}⟩|\le \Vert \mathbf{x}\Vert \cdot \Vert \mathbf{y}\Vert .$$

Proof. The result is trivial if $\mathbf{x}=\mathbf{0}$ or $\mathbf{y}=\mathbf{0}$.

Suppose $\mathbf{x}\ne \mathbf{0}$ and $\mathbf{y}\ne \mathbf{0}$. We have

$$\Vert \mathbf{x}+\lambda \mathbf{y}{\Vert }^2\ge 0$$

for all $\lambda \in \mathbb{R}$ since it's the square of a real number. Thus

$$⟨\mathbf{x}+\lambda \mathbf{y},\mathbf{x}+\lambda \mathbf{y}⟩\ge 0.$$

Expanding the left-hand side using the properties of scalar product, we have

$${\lambda }^2\Vert \mathbf{y}{\Vert }^2+2\lambda ⟨\mathbf{x},\mathbf{y}⟩+\Vert \mathbf{x}{\Vert }^2\ge 0.$$

The left-hand side is a quadratic polynomial in $\lambda$ with a positive coefficient of ${\lambda }^2$. If this inequality should hold for any real value of $\lambda$, the discriminant of this quadratic polynomial should be non-positive:

$$4{⟨\mathbf{x},\mathbf{y}⟩}^2-4\Vert \mathbf{x}{\Vert }^2\Vert \mathbf{y}{\Vert }^2\le 0$$

which rearranges to

$$|⟨\mathbf{x},\mathbf{y}⟩|\le \Vert \mathbf{x}\Vert \Vert \mathbf{y}\Vert .$$

■

The Cauchy-Schwarz inequality plays an important role in many fields in mathematics, including statistics.

Definition (Angle between two vectors in ${\mathbb{R}}^n$)

Let $\mathbf{x},\mathbf{y}\in {\mathbb{R}}^n$ be non-zero vectors. The angle between $\mathbf{x}$ and $\mathbf{y}$ is defined to be that angle $\theta$ such that

$$\cos \theta =\frac{⟨\mathbf{x},\mathbf{y}⟩}{\Vert \mathbf{x}\Vert \cdot \Vert \mathbf{y}\Vert }.$$

From the Cauchy-Schwarz inequality, we can see that the right-hand side in the above definition is bounded between $-1$ and $1$ so that we indeed have $-1\le \cos \theta \le 1$ as it should be.

Theorem (Triangle inequality)

Let $\mathbf{x},\mathbf{y}\in {\mathbb{R}}^n$. Then

$$\Vert \mathbf{x}+\mathbf{y}\Vert \le \Vert \mathbf{x}\Vert +\Vert \mathbf{y}\Vert .$$

Proof. We can use the Cauchy-Schwarz inequality to show the following.

$$\begin{array}{rcl}\Vert \mathbf{x}+\mathbf{y}{\Vert }^2& =& ⟨\mathbf{x}+\mathbf{y},\mathbf{x}+\mathbf{y}⟩\\ & =& \Vert \mathbf{x}{\Vert }^2+2⟨\mathbf{x},\mathbf{y}⟩+\Vert \mathbf{y}{\Vert }^2\\ & \le & \Vert \mathbf{x}{\Vert }^2+2|⟨\mathbf{x},\mathbf{y}⟩|+\Vert \mathbf{y}{\Vert }^2\\ & \le & \Vert \mathbf{x}{\Vert }^2+2\Vert \mathbf{x}\Vert \Vert \mathbf{y}\Vert +\Vert \mathbf{y}{\Vert }^2\\ & =& (\Vert \mathbf{x}\Vert +\Vert \mathbf{y}\Vert {)}^2.\end{array}$$

■