23.1. Random Vectors

Vectors and matrices give us a compact way of referring to random sequences like $X_1,X_2,\dots ,X_n$. The algebra of vectors and matrices gives us powerful tools for studying linear combinations of random variables.

In this section we will develop matrix notation for random sequences and then express familiar consequences of linearity of expectation and bilinearity of covariance in matrix notation. The probability theory in this section is not new – it consists of expectation and covariance facts that you have known for some time. But the representation is new and leads us to new insights.

A vector valued random variable, or more simply, a random vector, is a list of random variables defined on the same space. We will think of it as an $n\times 1$ column vector.

$$\begin{array}{r}\mathbf{X}=\left[\begin{array}{c}{X}_1\\ X_2\\ ⋮\\ X_n\end{array}\right]\end{array}$$

For ease of display, we will sometimes write $\mathbf{X}=[X_1{X}_2\dots X_n{]}^T$ where ${\mathbf{M}}^T$ is notation for the transpose of the matrix $\mathbf{M}$.

The mean vector of $\mathbf{X}$ is $\mathit{\mu }=[{\mu }_1{\mu }_2\dots {\mu }_n{]}^T$ where ${\mu }_i=E(X_i)$.

The covariance matrix of $\mathbf{X}$ is the $n\times n$ matrix $\mathbf{\Sigma }$ whose $(i,j)$ element is $Cov(X_i,X_j)$.

The $i$th diagonal element of $\mathbf{\Sigma }$ is the variance of $X_i$. The matrix is symmetric because of the symmetry of covariance.

Quick Check

A random vector $\left[\begin{array}{c}X\\ Y\end{array}\right]$ has mean vector $\left[\begin{array}{c}0\\ 0\end{array}\right]$ and covariance matrix $\left[\begin{array}{cc}1& 2.4\\ ?& 9\end{array}\right]$.

Which (if any) of $X$ and $Y$ are in standard units? Pick one option.

(i) Only $X$

(ii) Only $Y$

(iii) Both $X$ and $Y$

(iv) Neither $X$ nor $Y$

(v) There is not enough information to answer.

Answer

(i)

Quick Check

(Continuing the Quick Check above) Fill in the ? in the covariance matrix.

Answer

$2.4$

Quick Check

(Continuing the Quick Check above) Find the correlation between $X$ and $Y$.

Answer

$0.8$

Click to show

23.1.1. Linear Transformation: Mean Vector

Let $\mathbf{A}$ be an $m\times n$ numerical matrix and $\mathbf{b}$ an $m\times 1$ numerical vector. Consider the $m\times 1$ random vector $\mathbf{Y}=\mathbf{AX}+\mathbf{b}$.

We will call this a “linear transformation” of $\mathbf{X}$ though in fact it is an affine transformation, that is, a composition of the linear transformation $\mathbf{AX}$ and the translation by $\mathbf{b}$.

This representation gives us a compact way to describe multiple linear combinations of $\mathbf{X}$ simultaneously. For example, if $\mathbf{b}=[000{]}^T$ and

$$\begin{array}{r}\mathbf{A}=\left[\begin{array}{cccccc}1& 0& 0& 0& \cdots & 0\\ 1& -1& 0& 0& \cdots & 0\\ \frac{1}{n}& \frac{1}{n}& \frac{1}{n}& \frac{1}{n}& \cdots & \frac{1}{n}\end{array}\right]\end{array}$$

then

$$\begin{array}{r}\mathbf{AX}+\mathbf{b}=\left[\begin{array}{c}{X}_1\\ X_1-X_2\\ {\overline{X}}_n\end{array}\right]\end{array}$$

In general, if $\mathbf{Y}=\mathbf{AX}+\mathbf{b}$ then the $i$th element of $\mathbf{Y}$ is

$$Y_i={\mathbf{A}}_{i\ast }\mathbf{X}+\mathbf{b}(i)$$

where ${\mathbf{A}}_{i\ast }$ denotes the $i$th row of $\mathbf{A}$ and $\mathbf{b}(i)$ denotes the $i$th element of $\mathbf{b}$. Written longhand,

$$Y_i=a_{i1}{X}_1+a_{i2}{X}_2+\cdots +a_{in}{X}_n+b_i$$

where $a_{ij}$ is the $(i,j)$ entry of $\mathbf{A}$ and $b_i=\mathbf{b}(i)$.

Thus $Y_i$ is a linear combination of the elements of $\mathbf{X}$. Therefore by linearity of expectation,

$$E(Y_i)={\mathbf{A}}_{i\ast }\mathit{\mu }+\mathbf{b}(i)$$

Let ${\mathit{\mu }}_{\mathbf{Y}}$ be the mean vector of $\mathbf{Y}$. Then by the calculation above,

$${\mathit{\mu }}_{\mathbf{Y}}=\mathbf{A}\mathit{\mu }+\mathbf{b}$$

Click to show

23.1.2. Linear Transformation: Covariance Matrix

$Cov(Y_i,Y_j)$ can be calculated using bilinearity of covariance.

$$\begin{array}{r}\begin{array}{rl}Cov(Y_i,Y_j)& =Cov({\mathbf{A}}_{i\ast }\mathbf{X},{\mathbf{A}}_{j\ast }\mathbf{X})\\ & =Cov(\sum _{k=1}^n{a}_{ik}{X}_k,\sum _{l=1}^n{a}_{jl}{X}_l)\\ & =\sum _{k=1}^n\sum _{l=1}^n{a}_{ik}{a}_{jl}Cov(X_k,X_l)\\ & =\sum _{k=1}^n\sum _{l=1}^n{a}_{ik}Cov(X_k,X_l)t_{lj}\text{where }{t}_{lj}={\mathbf{A}}^T(l,j)\end{array}\end{array}$$

This is the $(i,j)$ element of $\mathbf{A}\mathbf{\Sigma }{\mathbf{A}}^T$. So if ${\mathbf{\Sigma }}_{\mathbf{Y}}$ denotes the covariance matrix $\mathbf{Y}$, then

$${\mathbf{\Sigma }}_{\mathbf{Y}}=\mathbf{A}\mathbf{\Sigma }{\mathbf{A}}^T$$

Click to show

Let us see what this formula implies for the variance of a single component of $\mathbf{Y}$.

Any component of $\mathbf{Y}$ is a linear combination of the elements of $\mathbf{X}$ and hence can be written as $\mathbf{aX}+b$ for some $1\times n$ vector $\mathbf{a}$ and some real number $b$.

The variance of this component of $\mathbf{Y}$ is a diagonal element of ${\mathbf{\Sigma }}_{\mathbf{Y}}$. By our calculation above, the diagonal element is equal to $\mathbf{a}\mathbf{\Sigma }{\mathbf{a}}^T$.

23.1.3. Constraints on $\mathbf{\Sigma }$

We know that $\mathbf{\Sigma }$ has to be symmetric. Also, no matter what $\mathbf{A}$ is, the elements on the main diagonal of ${\mathbf{\Sigma }}_{\mathbf{Y}}$ must all be non-negative as they are the variances of the elements of $\mathbf{Y}$.

By the observation above, this implies

$$\mathbf{a}\mathbf{\Sigma }{\mathbf{a}}^T\ge 0\text{for all }1\times n\text{ vectors }\mathbf{a}$$

That is, $\mathbf{\Sigma }$ must be positive semidefinite.

Usually, we will be working with covariance matrices that are positive definite, defined by

$$\mathbf{a}\mathbf{\Sigma }{\mathbf{a}}^T>0\text{for all }1\times n\text{ vectors }\mathbf{a}$$

The reason is that if $\mathbf{a}\mathbf{\Sigma }{\mathbf{a}}^T=0$ for some $\mathbf{a}$, then the linear transformation $\mathbf{aX}$ has variance $0$ and hence is a constant. Hence you can write some of the elements of $\mathbf{X}$ as linear combinations of the others and just study a reduced set of elements.