Expectations. Consider a matrix of random variables begin{equation*} mathbf{U=}left( begin{array}{cccc} u_{11} & u_{12} & cdots & u_{1c} \ u_{21} & u_{22} & cdots & u_{2c} \ vdots & vdots & ddots & vdots \ u_{n1} & u_{n2} & cdots & u_{nc} end{array} right). end{equation*} When we write the expectation of a matrix, this is short-hand for the matrix of expectations. Specifically, suppose that the joint probability function of ({u_{11}, u_{12}, …, u_{1c}, …, u_{n1}, …, u_{nc}}) is available to define the expectation operator. Then we define begin{equation*} mathrm{E} ~ mathbf{U} = left( begin{array}{cccc} mathrm{E }u_{11} & mathrm{E }u_{12} & cdots & mathrm{E }u_{1c} \ mathrm{E }u_{21} & mathrm{E }u_{22} & cdots & mathrm{E }u_{2c} \ vdots & vdots & ddots & vdots \ mathrm{E }u_{n1} & mathrm{E }u_{n2} & cdots & mathrm{E }u_{nc} end{array} right). end{equation*} As an important special case, consider the joint probability function for the random variables (y_1, ldots, y_n) and the corresponding expectations operator. Then begin{equation*} mathrm{E}~ mathbf{y=} mathrm{E } left( begin{array}{cccc} y_1 \ vdots \ y_n end{array} right) = left( begin{array}{cccc} mathrm{E }y_1 \ vdots \ mathrm{E }y_n end{array} right). end{equation*} By the linearity of expectations, for a non-random matrix A and vector B, we have E (A y + B) = A E y + B.
Variances. We can also work with second moments of random vectors. The variance of a vector of random variables is called the variance-covariance matrix. It is defined by begin{equation}label{E2:MatrixVar} mathrm{Var} ~ mathbf{y} = mathrm{E} ( (mathbf{y} – mathrm{E} mathbf{y})(mathbf{y} – mathrm{E} mathbf{y})^{prime} ). end{equation} That is, we can express begin{equation*} mathrm{Var}~mathbf{y=} mathrm{E } left( left( begin{array}{c} y_1 -mathrm{E } y_1 \ vdots \ y_n -mathrm{E } y_n end{array}right) left(begin{array}{ccc} y_1 – mathrm{E } y_1 & cdots & y_n – mathrm{E } y_n end{array}right) right) end{equation*} begin{equation*} = left( begin{array}{cccc} mathrm{Var}~y_1 & mathrm{Cov}(y_1, y_2) & cdots &mathrm{Cov}(y_1, y_n) \ mathrm{Cov}(y_2, y_1) & mathrm{Var}~y_2 & cdots & mathrm{Cov}(y_2, y_n) \ vdots & vdots & ddots & vdots\ mathrm{Cov}(y_n, y_1) & mathrm{Cov}(y_n, y_2) & cdots & mathrm{Var}~y_n \ end{array}right), end{equation*} because (mathrm{E} ( (y_i – mathrm{E} y_i)(y_j – mathrm{E} y_j) ) = mathrm{Cov}(y_i, y_j)) for (i neq j) and (mathrm{Cov}(y_i, y_i) = mathrm{Var}~y_i). In the case that (y_1, ldots, y_n) are mutually uncorrelated, we have that (mathrm{Cov}(y_i, y_j)=0) for (i neq j) and thus begin{equation*} mathrm{Var}~mathbf{y=} left( begin{array}{cccc} mathrm{Var}~y_1 & 0 & cdots & 0 \ 0 & mathrm{Var}~y_2 & cdots & 0 \ vdots & vdots & ddots & vdots\ 0 & 0 & cdots & mathrm{Var}~y_n \ end{array}right). end{equation*} Further, if the variances are identical so that (mathrm{Var}~y_i=sigma ^2), then we can write (mathrm{Var} ~mathbf{y} = sigma ^2 mathbf{I}), where I is the (n times n) identity matrix. For example, if (y_1, ldots, y_n) are i.i.d., then (mathrm{Var} ~mathbf{y} = sigma ^2 mathbf{I}).
From equation (2.10), it can be shown that begin{equation}label{E2:MatrixVarCalc} mathrm{Var}left( mathbf{Ay +B} right) = mathrm{Var}left( mathbf{Ay} right) = mathbf{A} left( mathrm{Var}~mathbf{y} right) mathbf{A}^{prime}. end{equation} For example, if (mathbf{A} = (a_1, a_2, ldots,a_n)= mathbf{a}^{prime}) and B = 0, then equation (2.11) reduces to begin{equation*} mathrm{Var}left( sum_{i=1}^n a_i y_i right) = mathrm{Var} left( mathbf{a^{prime} y} right) = mathbf{a^{prime}} left( mathrm{Var} ~mathbf{y} right) mathbf{a} = (a_1, a_2, ldots,a_n) left( mathrm{Var} ~mathbf{y} right) left(begin{array}{c} a_1 \ vdots \ a_n end{array}right) end{equation*} begin{equation*} = sum_{i=1}^n a_i^2 mathrm{Var} ~y_i ~+~2 sum_{i=2}^n sum_{j=1}^{i-1} a_i a_j mathrm{Cov}(y_i, y_j). end{equation*}
Definition – Multivariate Normal Distribution. A vector of random variables (mathbf{y} = left(y_1, ldots, y_n right)^{prime}) is said to be multivariate normal if all linear combinations of the form (sum_{i=1}^n a_i y_i) are normally distributed. In this case, we write (mathbf{y}sim N (mathbf{boldsymbol mu}, mathbf{Sigma} )), where (mathbf{boldsymbol mu} = mathrm{E}~ mathbf{y} ) is the expected value of y and (mathbf{Sigma}= mathrm{Var}~mathbf{y}) is the variance-covariance matrix of y. From the definition, we have that (mathbf{y}sim N (mathbf{boldsymbol mu}, mathbf{Sigma} )) implies that (mathbf{a^{prime}y}sim N (mathbf{a^{prime} boldsymbol mu}, mathbf{a^{prime}Sigma a} )). Thus, if (y_i) are i.i.d., then (sum_{i=1}^n a_i y_i) is distributed normally with mean (mu sum_{i=1}^n a_i ) and variance (sigma ^2 sum_{i=1}^n a_i ^2).