In matrix algebra, there is no concept of “division.” Instead, we extend the concept of “reciprocals” of real numbers. To begin, suppose that (mathbf{A}) is a square matrix of dimension (k times k) and let (mathbf{I}) be the (ktimes k) identity matrix. If there exists a (k times k) matrix (mathbf{B}) such that (mathbf{AB}=mathbf{I=BA}), then (mathbf{B}) is called the inverse of (mathbf{A}) and is written
begin{equation*} mathbf{B}=mathbf{A}^{-1}. end{equation*} Now, not all square matrices have inverses. Further, even when an inverse exists, it may not be easy to compute by hand. One exception to this rule are diagonal matrices. Suppose that (mathbf{A}) is diagonal matrix of the form
begin{equation*} mathbf{A=}left( begin{array}{ccc} a_{11} & cdots & 0 \ vdots & ddots & vdots \ 0 & cdots & a_{kk} end{array} right). text{ Then }mathbf{A}^{-1}mathbf{=}left( begin{array}{ccc} frac{1}{a_{11}} & cdots & 0 \ vdots & ddots & vdots \ 0 & cdots & frac{1}{a_{kk}} end{array} right). end{equation*} For example,
begin{equation*} begin{array}{cccc} left( begin{array}{cc} 2 & 0 \ 0 & -19 end{array} right) & left( begin{array}{cc} frac{1}{2} & 0 \ 0 & -frac{1}{19} end{array} right) & = & left( begin{array}{cc} 1 & 0 \ 0 & 1 end{array} right) \ mathbf{A} & mathbf{A}^{-1} & = & mathbf{I} end{array} . end{equation*} In the case of a matrix of dimension (2times 2), the inversion procedure can be accomplished by hand easily even when the matrix is not diagonal. In the (2times 2) case, we suppose that if
begin{equation*} mathbf{A=}left( begin{array}{cc} a & b \ c & d end{array} right), text{ then }mathbf{A}^{-1}mathbf{=}frac{1}{ad-bc}left( begin{array}{cc} d & -b \ -c & a end{array} right) text{.} end{equation*} Thus, for example, if
begin{equation*} mathbf{A=}left( begin{array}{cc} 2 & 2 \ 3 & 4 end{array} right) text{ then }mathbf{A}^{-1}mathbf{=}frac{1}{2(4)-2(3)} left( begin{array}{cc} 4 & -2 \ -3 & 2 end{array} right) =left( begin{array}{cc} 2 & -1 \ -3/2 & 1 end{array} right) text{.} end{equation*} As a check, we have
begin{align*} mathbf{Amathbf{A}^{-1}}&=left( begin{array}{cc} 2 & 2 \ 3 & 4 end{array} right) left( begin{array}{cc} 2 & -1 \ -3/2 & 1 end{array} right) =left( begin{array}{cc} 2(2)-2(3/2) & 2(-1)+2(1) \ 3(2)-4(3/2) & 3(-1)+4(1) end{array} right) \ &=left( begin{array}{cc} 1 & 0 \ 0 & 1 end{array} right) =mathbf{I}text{.} end{align*}
Basic Linear Regression Example of Matrix Inverses. With
begin{equation*} mathbf{X}^{prime }mathbf{X=}left( begin{array}{cc} n & sumlimits_{i=1}^{n}x_i \ sumlimits_{i=1}^{n}x_i & sumlimits_{i=1}^{n}x_i^2 end{array} right), end{equation*} we have
begin{equation*} left( mathbf{X}^{prime }mathbf{X}right) ^{-1}mathbf{=}frac{1}{nsum_{i=1}^{n}x_i^2-left( sum_{i=1}^{n}x_iright) ^2}left( begin{array}{cc} sumlimits_{i=1}^{n}x_i^2 & -sumlimits_{i=1}^{n}x_i \ -sumlimits_{i=1}^{n}x_i & n end{array} right). end{equation*} To simplify this expression, recall that (overline{x}=n^{-1} sum_{i=1}^{n}x_i). Thus,
begin{equation}label{E2:XPXInv} left( mathbf{X}^{prime }mathbf{X}right) ^{-1}mathbf{=}frac{1}{ sum_{i=1}^{n}x_i^2-noverline{x}^2}left( begin{array}{cc} n^{-1}sumlimits_{i=1}^{n}x_i^2 & -overline{x} \ -overline{x} & 1 end{array} right) . end{equation}
Section 3.1 will discuss the relation (mathbf{b}=left( mathbf{X}^{prime}mathbf{X}right)^{-1} mathbf{X}^{prime}mathbf{y}). To illustrate the calculation, we have
begin{eqnarray*} mathbf{b} &=&left( mathbf{X}^{prime }mathbf{X}right) ^{-1}mathbf{X} ^{prime }mathbf{y=}frac{1}{sum_{i=1}^{n}x_i^2-noverline{x}^2} left( begin{array}{cc} n^{-1}sumlimits_{i=1}^{n}x_i^2 & -overline{x} \ -overline{x} & 1 end{array} right) left( begin{array}{c} sumlimits_{i=1}^{n}y_i \ sumlimits_{i=1}^{n}x_iy_i end{array} right) \ &=&frac{1}{sum_{i=1}^{n}x_i^2-noverline{x}^2}left( begin{array}{c} sumlimits_{i=1}^{n}left( overline{y}x_i^2-overline{x} x_iy_iright) \ sumlimits_{i=1}^{n}x_iy_i-noverline{x}overline{y} end{array} right) =left( begin{array}{c} b_0 \ b_1 end{array} right) . end{eqnarray*} From this expression, we may see
begin{equation*} b_1=frac{sumlimits_{i=1}^{n}x_iy_i-noverline{x}overline{y}}{ sumlimits_{i=1}^{n}x_i^2-noverline{x}^2} end{equation*} and
begin{equation*} b_0=frac{overline{y}sumlimits_{i=1}^{n}x_i^2-overline{x} sumlimits_{i=1}^{n}x_iy_i}{sumlimits_{i=1}^{n}x_i^2-noverline{x} ^2}=frac{overline{y}left( sumlimits_{i=1}^{n}x_i^2-noverline{x} ^2right) -overline{x}left( sumlimits_{i=1}^{n}x_i y_i – noverline{x} overline{y}right) }{sumlimits_{i=1}^{n}x_i^2-noverline{x}^2}= overline{y}-b_1overline{x}. end{equation*} These are the usual expressions for the slope (b_1) (Exercise 2A.8) and intercept (b_0).