Any covariance matrix is positive semi-definite: A proof

-
Let \(\mathbf{X} = (X_1, X_2, \cdots, X_n)^\top \in \mathbb{R}^n\) be a vector of random variables. The covariance matrix \(\Sigma\) of \(\mathbf{X}\) is a square (\(n\times n\)) matrix whose elements are covariances between the components of \(\mathbf{X}\). That is,

\[\Sigma_{ij} = \mathrm{Cov}(X_i,X_j)\]

where \(\mathrm{Cov}(X_i,X_j)\) is the covariance between \(X_i\) and \(X_j\), \(i,j = 1, 2, \cdots, n\): 

\[\mathrm{Cov}(X_i, X_j) = \mathbb{E}[(X_i - \mathbb{E}[X_i])(X_j - \mathbb{E}[X_j])].\]

Here, \(\mathbb{E}[\cdot]\) indicates the expectation value of a random variable. Any covariance matrix has the following properties:

  1. Symmetric. That is, \[\Sigma = \Sigma^\top.\]
  2. Positive semi-definite. That is,\[\forall \mathbf{v} \in \mathbb{R}^n, \mathbf{v}^\top\Sigma\mathbf{v} \geq 0.\]

See also: Positive definite matrix (Wolfram MathWorld)

The symmetry is obvious from the definition of the covariance matrix. 

Now, let us prove that the covariance matrix is positive semi-definite. First, note that the covariance matrix can be expressed as
\[\Sigma = \int_{\mathbb{R}^n}\rho(\mathbf{x})(\mathbf{x} - \boldsymbol{\mu})(\mathbf{x}-\boldsymbol{\mu})^\top \, d\mathbf{x}\]
where \(\rho(\mathbf{x})\) is the density of \(\mathbf{X}\), and
\[\boldsymbol{\mu} = \mathbb{E}[\mathbf{X}] = \int_{\mathbb{R}^n}\rho(\mathbf{x})\mathbf{x} \,d\mathbf{x} \in \mathbb{R}^n\]
is the expectation value of \(\mathbf{X}\). 

Let \(\mathbf{v}\in \mathbb{R}^n\) be an arbitrary vector. We have
\[ \begin{eqnarray}\mathbf{v}^{\top}\Sigma\mathbf{v} &=& \int_{\mathbb{R}^n}\rho(\mathbf{x})\mathbf{v}^{\top}(\mathbf{x}-\boldsymbol{\mu})(\mathbf{x} - \boldsymbol{\mu})^{\top}\mathbf{v}\,d\mathbf{x}\\ &=& \int_{\mathbb{R}^n}\rho(\mathbf{x})[(\mathbf{x}-\boldsymbol{\mu})^{\top}\mathbf{v}]^{\top}[(\mathbf{x}-\boldsymbol{\mu})^{\top}\mathbf{v}]\,d\mathbf{x}\\ &=& \int_{\mathbb{R}^n}\rho(\mathbf{x})[(\mathbf{x}-\boldsymbol{\mu})^{\top}\mathbf{v}]^2\,d\mathbf{x} \geq 0 \end{eqnarray}\]
because the integrand is non-negative.

Remark (2022-10-14): In the initial version of this post, I used the eigenvalue decomposition of the covariance matrix to prove the positive semi-definiteness. But that is not only unnecessary but overly complicated. The above proof is simpler and more direct.

Comments

Post a Comment

Popular posts from this blog

Applications of multiple integrals

-
Image
We can use multiple integrals to compute areas and volumes of various shapes. Area of a planar region Definition (Area) Let \(D\) be a bounded closed region in \(\mathbb{R}^2\). \(D\) is said to have an area if the multiple integral of the constant function 1 over \(D\), \(\iint_Ddxdy\), exists. Its value is denoted by \(\mu(D)\): \[\mu(D) = \iint_Ddxdy.\] Example . Let us calculate the area of the disk \(D = \{(x,y)\mid x^2 + y^2 \leq a^2\}\). Using the polar coordinates, \(x = r\cos\theta, y = r\sin\theta\), \(dxdy = rdrd\theta\), and the ranges of \(r\) and \(\theta\) are \([0, a]\) and \([0, 2\pi]\), respectively. Thus, \[\begin{eqnarray*} \mu(D) &=& \iint_Ddxdy\\ &=&\int_0^a\left(\int_0^{2\pi}rd\theta\right)dr\\ &=&2\pi\int_0^a rdr\\ &=&2\pi\left[\frac{r^2}{2}\right]_0^a = \pi a^2. \end{eqnarray*}\] □ Volume of a solid figure Definition (Volume) Let \(V\) be a solid figure in the \((x,y,z)\) space \(\mathbb{R}^3\). \(V\) is...
Read more

Birth process

-
Image
  Defining the birth process  Consider a colony of bacteria that never dies. We study the following process known as  the birth process , also known as  the Yule process . The colony starts with \(n_0\) cells at time \(t = 0\). Assume that the probability that any individual cell divides in the time interval \((t, t + \delta t)\) is proportional to \(\delta t\) for small \(\delta t\). Further assume that each cell division is independent of others. Let \(\lambda\) be the  birth rate.  The probability of a cell division for a population of \(n\) cells during \(\delta t\) is \(\lambda n \delta t\). We assume that the probability that two or more births take place in the time interval \(\delta t\) is \(o(\delta t)\). That is, it can be ignored. Consequently, the probability that no cell divides during \(\delta t\) is \(1 - \lambda n \delta t - o(\delta t)\). Note that this process is an example of the Markov chain with states \({n_0}, {n_0 + 1}, {n_0 + 2}...
Read more

Improper multiple integrals

-
Image
Consider integrating a function \(f(x,y)\) over a region \(D\) which may not be bounded or closed. In the case of a univariate function, this corresponds to the improper integral where we took the limits of the endpoints of a closed interval. In the case of multiple integrals, we adopt the notion of a "sequence of regions." Consider a sequence of regions \(\{K_n\}\) where each \(K_n\) is a subset of \(\mathbb{R}^2\) that satisfies the following conditions: (a) \(K_1 \subset K_2\)\(\subset \cdots \subset\) \(K_n \subset K_{n+1} \subset \cdots\). (b) For all \(n\in \mathbb{N}\), \(K_n \subset D\). (c) For all \(n \in\mathbb{N}\), \(K_n\) is bounded and closed. (d) For any bounded closed set \(F\) that is included in \(D\) (i.e., \(F \subset D\)), if \(n\) is sufficiently large, then \(F \subset K_n\).  In other words: for all bounded closed \(F \subset D\), there exists some \(N\in \mathbb{N}\) such that, for all \(n\in \mathbb{N}\), if \(n \geq N\) then \(F \subset K_...
Read more