19 Representing Linear Regression with Linear Algebra
The last few chapters were a lot of (necessary) maths, but you made it. It is now time to bring linear algebra and linear regression together. As their names indicate, the two should already have a lot in common.
The goal is to represent a linear regression with the simple expression:
\[ \hat{\mathbf{y}} = \mathbf{X}\mathbf{w} \]
With:
- \(\hat{\mathbf{y}}\): a vector of predictions
- \(\mathbf{X}\): the feature matrix
- \(\mathbf{w}\): the vector of coefficients
You can recognise the notation introduced in previous chapters. Matrices are noted as bolded capital letters, and vectors as bolded lowercase letters.
19.1 From Calculations to Dot Products
Let’s use the example explored earlier:
\[ \text{Predicted Ice Cream Sales} = 2 + 20 \times \text{Temperature} \]
The forecasted temperatures in the next three days are: \(20\), \(21\), and \(22\).
When generating predictions for these three points, we need to calculate the following:
\[\begin{aligned} \hat{\text{Sales}}_{\text{day 1}} &= 2 + 20 \times 20 = 402 \\ \hat{\text{Sales}}_{\text{day 2}} &= 2 + 20 \times 21 = 422 \\ \hat{\text{Sales}}_{\text{day 3}} &= 2 + 20 \times 22 = 442 \end{aligned}\]
There is some regularity and symmetry in the calculations above. This is the first sign that we may be able to use linear algebra.
The symmetries may become even more apparent when noting the prediction calculations as:
\[\begin{aligned} \hat{\text{Sales}}_{\text{day 1}} &= 1 \times 2 + 20 \times 20 = 402 \\ \hat{\text{Sales}}_{\text{day 2}} &= 1 \times 2 + 21 \times 20 = 422 \\ \hat{\text{Sales}}_{\text{day 3}} &= 1 \times 2 + 22 \times 20 = 442 \end{aligned}\]
If you think that this starts to look like the dot product of two vectors, I am very proud of you. For readers who do not remember what a dot product is, I would recommend rereading the previous chapter.
Noting the prediction calculations as dot products:
\[\begin{aligned} \hat{\text{Sales}}_{\text{day 1}} &= (1, 20) \cdot (2, 20) = 1 \times 2 + 20 \times 20 = 402 \\ \hat{\text{Sales}}_{\text{day 2}} &= (1, 21) \cdot (2, 20) = 1 \times 2 + 21 \times 20 = 422 \\ \hat{\text{Sales}}_{\text{day 3}} &= (1, 22) \cdot (2, 20) = 1 \times 2 + 22 \times 20 = 442 \end{aligned}\]
We notice that one of the vectors \((2, 20)\) never changes. This vector corresponds to \(\mathbf{w}\), the coefficient vector:
\[ \mathbf{w} = \begin{pmatrix} 2 \\ 20 \end{pmatrix} \]
It contains the intercept and the slope.
Looking at the other vectors:
\[ (1, 20), \quad (1, 21), \quad (1, 22) \]
We see that the first component is always a \(1\). The second component of the vector is the forecasted temperature for the days to come.
19.2 From Dot Products to Matrix Multiplication
We still see regularity and symmetry in these calculations. Another sign that we can use more linear algebra. What if we could do all these predictions at once?
We can do so by stacking all the data vectors together in a data matrix:
\[ \mathbf{X} = \begin{pmatrix} 1 & 20 \\ 1 & 21 \\ 1 & 22 \end{pmatrix} \]
By multiplying this matrix \(\mathbf{X}\) with the coefficient vector \(\mathbf{w}\), we get:
\[ \mathbf{X}\mathbf{w} = \begin{pmatrix} 1 & 20 \\ 1 & 21 \\ 1 & 22 \end{pmatrix} \begin{pmatrix} 2 \\ 20 \end{pmatrix} = \begin{pmatrix} 1 \times 2 + 20 \times 20 \\ 1 \times 2 + 21 \times 20 \\ 1 \times 2 + 22 \times 20 \end{pmatrix} = \begin{pmatrix} 402 \\ 422 \\ 442 \end{pmatrix} = \hat{\mathbf{y}} \]
This final vector \(\hat{\mathbf{y}}\) is our prediction vector. It contains an ice cream sales prediction for each of the rows of the matrix. We can now represent a linear regression with:
\[ \hat{\mathbf{y}} = \mathbf{X}\mathbf{w} \]
19.3 Extending to Multiple Linear Regression
It is now time to get back to multiple linear regression. As a quick reminder, a multiple linear regression generates predictions of a target variable through a combination of features:
\[ \text{Ice Cream Sales} = 10 + 25 \times \text{Temperature} - 3 \times \text{Rainfall} \]
The table below shows three observations with temperature, rainfall, and predicted ice cream sales:
| Day | Temperature (°C) | Rainfall (mm) | Predicted Sales |
|---|---|---|---|
| 1 | 20 | 5 | 495 |
| 2 | 21 | 10 | 505 |
| 3 | 22 | 2 | 554 |
We already introduced the dot and sigma notations in a previous chapter. But can we do better?
Luckily for us, no additional notation is necessary. We simply need to extend the vector \(\mathbf{w}\) to include a coefficient for rainfall:
\[ \mathbf{w} = \begin{pmatrix} 10 \\ 25 \\ -3 \end{pmatrix} \]
And extend the \(\mathbf{X}\) matrix to include a new column for rainfall:
\[ \mathbf{X} = \begin{pmatrix} 1 & 20 & 5 \\ 1 & 21 & 10 \\ 1 & 22 & 2 \end{pmatrix} \]
Just like this, we can generate a vector of predictions:
\[\begin{aligned} \hat{\mathbf{y}} = \mathbf{X}\mathbf{w} &= \begin{pmatrix} 1 & 20 & 5 \\ 1 & 21 & 10 \\ 1 & 22 & 2 \end{pmatrix} \begin{pmatrix} 10 \\ 25 \\ -3 \end{pmatrix} \\ &= \begin{pmatrix} 1 \times 10 + 20 \times 25 + 5 \times (-3) \\ 1 \times 10 + 21 \times 25 + 10 \times (-3) \\ 1 \times 10 + 22 \times 25 + 2 \times (-3) \end{pmatrix} = \begin{pmatrix} 495 \\ 505 \\ 554 \end{pmatrix} \end{aligned}\]
Exercise 19.1 A multiple linear regression predicts house prices using three features:
\[ \text{Price} = 50 + 3 \times \text{Size} + 10 \times \text{Rooms} - 2 \times \text{Age} \]
Write out the \(\mathbf{X}\) matrix and \(\mathbf{w}\) vector for the following three houses, and compute \(\hat{\mathbf{y}} = \mathbf{X}\mathbf{w}\):
| House | Size (m²) | Rooms | Age (years) |
|---|---|---|---|
| A | 60 | 3 | 10 |
| B | 80 | 4 | 5 |
| C | 100 | 5 | 20 |
19.4 Going Beyond Two Features
We do not have to stop there. With the notation:
\[ \hat{\mathbf{y}} = \mathbf{X}\mathbf{w} \]
We can accommodate any finite number of features. For a multiple linear regression with \(k\) features and \(n\) observations:
\[ \hat{\mathbf{y}} = \mathbf{X}\mathbf{w} = \begin{pmatrix} 1 & x_{1,1} & x_{1,2} & \cdots & x_{1,k} \\ 1 & x_{2,1} & x_{2,2} & \cdots & x_{2,k} \\ \vdots & \vdots & \vdots & \ddots & \vdots \\ 1 & x_{n,1} & x_{n,2} & \cdots & x_{n,k} \end{pmatrix} \begin{pmatrix} w_0 \\ w_1 \\ w_2 \\ \vdots \\ w_k \end{pmatrix} \]
With this approach, we can concisely represent a multiple linear regression with any number of features. Whether we are predicting ice cream sales from \(2\) features or house prices from \(20\), the notation remains the same:
\[ \hat{\mathbf{y}} = \mathbf{X}\mathbf{w} \]
19.5 Final Thoughts
It took us a lot of maths to get here. By introducing all this notation, are we not falling into the typical mathematical trap of abstraction?
For now, you have had to believe me that all of these mathematics were worth learning. Whether predicting with \(1\) feature or \(100\), the entire model can be written in just two characters: \(\mathbf{X}\mathbf{w}\).
Linear regression is now as easy as a matrix multiplication. In the next chapter we will fit this regression to historical data.
19.6 Solutions
Solution 19.1. Exercise 19.1
The coefficient vector:
\[ \mathbf{w} = \begin{pmatrix} 50 \\ 3 \\ 10 \\ -2 \end{pmatrix} \]
The feature matrix (with a column of \(1\)s for the intercept):
\[ \mathbf{X} = \begin{pmatrix} 1 & 60 & 3 & 10 \\ 1 & 80 & 4 & 5 \\ 1 & 100 & 5 & 20 \end{pmatrix} \]
Computing \(\hat{\mathbf{y}} = \mathbf{X}\mathbf{w}\):
\[\begin{aligned} \hat{y}_A &= 1 \times 50 + 60 \times 3 + 3 \times 10 + 10 \times (-2) = 50 + 180 + 30 - 20 = 240 \\ \hat{y}_B &= 1 \times 50 + 80 \times 3 + 4 \times 10 + 5 \times (-2) = 50 + 240 + 40 - 10 = 320 \\ \hat{y}_C &= 1 \times 50 + 100 \times 3 + 5 \times 10 + 20 \times (-2) = 50 + 300 + 50 - 40 = 360 \end{aligned}\]
\[ \hat{\mathbf{y}} = \begin{pmatrix} 240 \\ 320 \\ 360 \end{pmatrix} \]