Norm and scalar product

-

 A vector space is a set with addition and scalar products. We can introduce more structures into this set. Here, we define a vector's norm (length) and the scalar product (dot product) between vectors.



Definition (Norm, length, modulus)

The norm (also called length or modulus) of the vector \(\mathbf{x} = (x_1,\cdots, x_n)\) is the real number defined by

\[\|\mathbf{x}\| = \sqrt{\sum_{i=1}^{n}x_i^2} = \sqrt{x_1^2 + x_2^2 + \cdots + x_n^2}.\tag{eq:norm}\]

Remark. The norm of a vector, as defined above, is also known as the 2-norm or the Euclidean norm of the vector. More generally, for \(k\in\mathbb{N}\), the \(k\)-norm of the vector \(\mathbf{x}\in\mathbb{R}^n\) is defined as

\[\|\mathbf{x}\|_k = \sqrt[k]{\sum_{i=1}^{n}|x_i|^k}.\]

□

Example. When \(n = 1\), the norm of \(\mathbf{x} = (x_1)\) is given by

\[\|\mathbf{x}\| = \sqrt{x_1^2} = |x_1|\]

which coincides with the absolute value of the real number \(x_1\).

When \(n=2\), the norm of of \(\mathbf{x} = (x_1, x_2)\) is

\[\|\mathbf{x}\| = \sqrt{x_1^2 + x_2^2}\]

which is the distance between the origin and \(\mathbf{x} = (x_1, x_2)\) in the 2-dimensional space. 

We can consider the modulus of a vector defined by (eq:norm) as a generalization of the distance between the origin and \(\mathbf{x}\) in the \(n\)-dimensional space. □

Definition (Unit vector)

Let \(\mathbf{x}\in\mathbb{R}^n\). \(\mathbf{x}\) is called a unit vector if \(\|\mathbf{x} \| = 1\). 


We can define a ``multiplication'' between two vectors to obtain a scalar.

Definition (Scalar product)

Let \(\mathbf{x} = (x_1,\cdots,x_n)\), \(\mathbf{y} = (y_1,\cdots,y_n)\in\mathbb{R}^n\). The scalar product of these two vectors is defined by

\[\braket{\mathbf{x}, \mathbf{y}} = \sum_{i=1}^{n}x_iy_i = x_1y_1 + x_2y_2 + \cdots + x_ny_n.\label{eq:dotprod}\]

Remark. The scalar product is also called the inner product or dot product, and they can be denoted as \(\mathbf{x}\cdot\mathbf{y}\) or \((\mathbf{x},\mathbf{y})\) or \(\braket{\mathbf{x}|\mathbf{y}}\). □

Remark. The scalar product may be considered as a map

\[\braket{\cdot , \cdot} : \mathbb{R}^n\times \mathbb{R}^n \to \mathbb{R}\]

that maps a pair of vectors to a real number. □

You should prove the following lemma.

Lemma

Let \(\mathbf{x}, \mathbf{y}, \mathbf{z}\in\mathbb{R}^n\) and \(\lambda \in \mathbb{R}\).
  1. \(\|\mathbf{x}\|^2 = \braket{\mathbf{x},\mathbf{x}}\).
  2. \(\|\lambda \mathbf{x}\| = |\lambda|\cdot\|\mathbf{x}\|\).
  3. \(\braket{\mathbf{x},\mathbf{y}} = \braket{\mathbf{y},\mathbf{x}}\).
  4. \(\braket{\mathbf{x}+\mathbf{y},\mathbf{z}} = \braket{\mathbf{x},\mathbf{z}} + \braket{\mathbf{y},\mathbf{z}}\).
  5. \(\braket{\mathbf{x}, \mathbf{y}+\mathbf{z}} = \braket{\mathbf{x},\mathbf{y}} + \braket{\mathbf{x},\mathbf{z}}\).
  6. \(\lambda\braket{\mathbf{x},\mathbf{y}} = \braket{\lambda\mathbf{x},\mathbf{y}}\).
  7. \(\lambda\braket{\mathbf{x},\mathbf{y}} = \braket{\mathbf{x},\lambda\mathbf{y}}\).
The next theorem provides a geometric interpretation of the scalar product.

Theorem (Angle between vectors in \(\mathbb{R}^2\))

Suppose that \(\mathbf{u}, \mathbf{v}\in\mathbb{R}^2\), then 

\[\braket{\mathbf{u},\mathbf{v}} = \|\mathbf{u}\|\cdot\|\mathbf{v}\|\cdot\cos\theta\]

where \(\theta\) is the angle between the vectors \(\mathbf{u}\) and \(\mathbf{v}\).

Proof. First, consider the case where \(\mathbf{u}\) and \(\mathbf{v}\) are unit vectors, namely, \(\|\mathbf{u}\| = \|\mathbf{v}\| = 1\).

Then we can write \(\mathbf{u} = (u_1, u_2) = (\cos\phi, \sin\phi)\) and \(\mathbf{v} = (v_1, v_2) = (\cos\psi, \sin\psi)\) where \(\phi\) and \(\psi\) are angles of \(\mathbf{u}\) and \(\mathbf{v}\) from \((1,0)\), respectively. Then

\[\braket{\mathbf{u},\mathbf{v}} = \cos\phi\cos\psi + \sin\phi\sin\psi = \cos(\phi - \psi).\]

Note that \(\phi - \psi\) is the angle between \(\mathbf{u}\) and \(\mathbf{v}\) so that \(\theta = \phi - \psi\), and the claim holds.


Next, consider the general case where \(\mathbf{u}\) and \(\mathbf{v}\) are not unit vectors. We can write \(\mathbf{u} = \|\mathbf{u}\|\hat{\mathbf{u}}\) and \(\mathbf{v} = \|\mathbf{v}\|\hat{\mathbf{v}}\) where \(\hat{\mathbf{u}}\) and \(\hat{\mathbf{v}}\) are unit vectors. Thus,

\[\braket{\mathbf{u},\mathbf{v}} = \|\mathbf{u}\|\cdot\|\mathbf{v}\|\braket{\hat{\mathbf{u}},\hat{\mathbf{v}}} = \|\mathbf{u}\|\cdot\|\mathbf{v}\|\cos\theta\]

as required. ■

Remark. Let's look at this theorem from a different perspective. Recall that we constructed \(\mathbb{C}\) from \(\mathbb{R}^2\). 

See also: Constructing complex numbers.

So, a complex number is actually a vector in \(\mathbb{R}^2\). The unit vector \({\mathbf{u}}\) corresponds to the complex number \(e^{i\phi}\), and the unit vector \({\mathbf{v}}\) corresponds to \(e^{i\psi}\).

Note that the scalar product

\[\braket{\mathbf{u},\mathbf{v}} = \braket{(u_1,u_2), (v_1,v_2)} = u_1v_1 + u_2v_2\]

is the real part of 

\[(u_1 - iu_2)(v_1 + iv_2) = (u_1v_1 + u_2v_2) + i(u_1v_2 - u_2v_1),\]

which, in turn, is the real part of

\[e^{-i\phi}e^{i\psi} = e^{-i(\phi-\psi)} = \cos(\phi-\psi) - i\sin(\phi-\psi).\]

□

We can prove the same theorem for \(\mathbb{R}^3\) (You should try to prove it). We want the same result for any dimension, \(\mathbb{R}^n\). How do we prove that? No, we don't prove that. Instead, we define the angle between two vectors in terms of the scalar product. But before that, we need the following theorem.

Theorem (Cauchy-Schwarz inequality)

Let \(\mathbf{x},\mathbf{y}\in\mathbb{R}^n\). Then

\[|\braket{\mathbf{x},\mathbf{y}}| \leq \|\mathbf{x}\|\cdot\|\mathbf{y}\|.\]

Proof. The result is trivial if \(\mathbf{x} = \mathbf{0}\) or \(\mathbf{y} = \mathbf{0}\).


Suppose \(\mathbf{x} \neq \mathbf{0}\) and \(\mathbf{y} \neq \mathbf{0}\). We have

\[\|\mathbf{x} + \lambda \mathbf{y}\|^2 \geq 0\]

for all \(\lambda \in \mathbb{R}\) since it's the square of a real number. Thus

\[\braket{\mathbf{x} + \lambda \mathbf{y}, \mathbf{x} + \lambda \mathbf{y}} \geq 0.\]

Expanding the left-hand side using the properties of scalar product, we have

\[\lambda^2\|\mathbf{y}\|^2 + 2\lambda\braket{\mathbf{x},\mathbf{y}} + \|\mathbf{x}\|^2 \geq 0.\]

The left-hand side is a quadratic polynomial in \(\lambda\) with a positive coefficient of \(\lambda^2\). If this inequality should hold for any real value of \(\lambda\), the discriminant of this quadratic polynomial should be non-positive:

\[4\braket{\mathbf{x},\mathbf{y}}^2 - 4\|\mathbf{x}\|^2\|\mathbf{y}\|^2 \leq 0\]

which rearranges to

\[|\braket{\mathbf{x},\mathbf{y}}| \leq \|\mathbf{x}\|\|\mathbf{y}\|.\]

■

The Cauchy-Schwarz inequality plays an important role in many fields in mathematics, including statistics.

Definition (Angle between two vectors in \(\mathbb{R}^n\))

Let \(\mathbf{x},\mathbf{y}\in\mathbb{R}^n\) be non-zero vectors. The angle between \(\mathbf{x}\) and \(\mathbf{y}\) is defined to be that angle \(\theta\) such that

\[\cos\theta = \frac{\braket{\mathbf{x},\mathbf{y}}}{\|\mathbf{x}\|\cdot\|\mathbf{y}\|}.\]

From the Cauchy-Schwarz inequality, we can see that the right-hand side in the above definition is bounded between \(-1\) and \(1\) so that we indeed have \(-1 \leq \cos\theta \leq 1\) as it should be.

Theorem (Triangle inequality)

Let \(\mathbf{x}, \mathbf{y}\in\mathbb{R}^n\). Then

\[\|\mathbf{x} + \mathbf{y}\| \leq \|\mathbf{x}\| + \|\mathbf{y}\|.\]

Proof. We can use the Cauchy-Schwarz inequality to show the following.

\[\begin{eqnarray*} \|\mathbf{x} + \mathbf{y}\|^2 &=& \braket{\mathbf{x} + \mathbf{y}, \mathbf{x} + \mathbf{y}}\\ &=& \|\mathbf{x}\|^2 + 2\braket{\mathbf{x},\mathbf{y}} + \|\mathbf{y}\|^2\\ &\leq&\|\mathbf{x}\|^2 + 2|\braket{\mathbf{x},\mathbf{y}}| + \|\mathbf{y}\|^2\\ &\leq&\|\mathbf{x}\|^2 + 2\|\mathbf{x}\|\|\mathbf{y}\| + \|\mathbf{y}\|^2\\ &=& (\|\mathbf{x}\| + \|\mathbf{y}\|)^2. \end{eqnarray*}\]

■

Comments

Post a Comment

Popular posts from this blog

Open sets and closed sets in \(\mathbb{R}^n\)

-
Image
Open sets In \(\mathbb{R}\), we have the notion of an open interval such as \((a, b) = \{x \in \mathbb{R} | a < x < b\}\). We want to extend this idea to apply to \(\mathbb{R}^n\). We also introduce the notions of bounded sets and closed sets in \(\mathbb{R}^n\). Recall that the \(\varepsilon\)-neighbor of a point \(x\in\mathbb{R}^n\) is defined as \(N_{\varepsilon}(x) = \{y \in \mathbb{R}^n | d(x, y) < \varepsilon \}\) where \(d(x,y)\) is the distance between \(x\) and \(y\). Definition (Open set) A subset \(U\) of \(\mathbb{R}^n\) is said to be an  open set  if the following holds: \[\forall x \in U ~ \exists \delta > 0 ~ (N_{\delta}(x) \subset U).\tag{Eq:OpenSet}\] That is, for every point in an open set \(U\), we can always find an open ball centered at that point, that is included in \(U\). See the following figure. Perhaps, it is instructive to see what is  not  an open set. Negating (Eq:OpenSet), we have \[\exists x \in U ~ \forall \delta > 0 ~ (N_{\delta}(x) \not
Read more

Euclidean spaces

-
Image
We would like to study multivariate functions (i.e., functions of many variables), continuous multivariate functions in particular. To define continuity, we need a measure of "closeness" between points. One measure of closeness is the Euclidean distance. The set \(\mathbb{R}^n\) (with \(n \in \mathbb{N}\)) with the Euclidean distance function is called a Euclidean space. This is the space where our functions of interest live. The real line is a geometric representation of \(\mathbb{R}\), the set of all real numbers. That is, each \(a \in \mathbb{R}\) is represented as the point \(a\) on the real line. The coordinate plane , or the \(x\)-\(y\) plane , is a geometric representation of \(\mathbb{R}^2\), the set of all pairs of real numbers. Each pair of real numbers \((a, b)\) is visualized as the point \((a, b)\) in the plane. Remark . Recall that \(\mathbb{R}^2 = \mathbb{R}\times\mathbb{R} = \{(x, y) | x, y \in \mathbb{R}\}\) is the Cartesian product of \(\mathbb{R}\) with i
Read more

Newton's method

-
Image
Newton's method (the Newton-Raphson method) is a very powerful numerical method for solving nonlinear equations.  Suppose we'd like to solve a non-linear equation \(f(x) = 0\), where \(f(x)\) is a (twice) differentiable non-linear function. Newton's method generates a sequence of numbers \(c_1, c_2, c_3, \cdots\) that converges to a solution of the equation. That is, if \(\alpha\) is a solution (i.e., \(f(\alpha) = 0\)) then,  \[\lim_{n\to\infty}c_n = \alpha,\] and this sequence \(\{c_n\}\) is generated by a series of linear approximations of the function \(f(x)\). Theorem (Newton's method) Let \(f(x)\) be a function that is twice differentiable on an open interval \(I\) that contains the closed interval \([a, b]\) (i.e., \([a,b]\subset I\)) and satisfy the following conditions: \(f(a) < 0\) and \(f(b) > 0\); For all \(x\in [a, b]\), \(f'(x) > 0\) and \(f''(x) > 0\). Let us define the sequence \(\{c_n\}\) by \[ \begin{eqnarray} c_1 &
Read more