10 The Chain Rule
The chain rule is the last concept we need to find the slope of a linear regression that minimises the mean squared error. It is a tool we use to compute the derivatives of complex expressions: functions of functions.
10.1 The Chain Rule
It can be tricky to find the derivative of:
\[ h(x) = (4x - 3)^2 \]
with regards to \(x\). It is possible, but would require a bit of work. From the previous chapter on derivatives, we have a few alternatives:
- Expand the expression, transforming it into a sum using \((a + b)^2 = a^2 + 2ab + b^2\)
- Use the product rule on \((4x - 3) \times (4x - 3)\)
Exercise 10.1 Expand \(h(x) = (4x - 3)^2\) into a sum to find the derivative of \(h(x)\) with regards to \(x\).
I have used \(h\) as a function name to keep \(f\) and \(g\) free for what comes next. The letters \(f\), \(g\), and \(h\) simply act as function names.
One easier way to differentiate:
\[ h(x) = (4x - 3)^2 \]
is to see it not as one, but two functions \(f\) and \(g\):
\[ h(x) = f(g(x)) \]
Where:
\[ f(u) = u^2 \quad \text{and} \quad g(x) = 4x - 3 \]
I have introduced a new variable \(u\) for clarity. Like \(x\), this is a variable that can take any number. In this scenario:
\[ f(g(x)) = (4x - 3)^2 = h(x) \]
The original question remains the same. How would \(h\) (or \(f(g)\)) change when \(x\) increases slightly? This can be decomposed into two questions:
- How will \(f\) change as \(g\) increases?
- How will \(g\) change as \(x\) increases?
I like to see this as dominos falling onto one another:
\[ x \text{ increases} \rightarrow g(x) \text{ changes} \rightarrow f(g) \text{ changes} \]
A change in \(x\) causes a change in \(g\), which in turn causes a change in \(f\). These mathematical dominos can be noted as:
\[ \frac{d}{dx} f(g(x)) = \frac{df}{dg} \times \frac{dg}{dx} \]
Read that as:
- \(\frac{d}{dx} f(g(x))\): the change of \(f(g(x))\) as \(x\) increases
- \(\frac{df}{dg}\): the change of \(f\) as \(g\) increases
- \(\frac{dg}{dx}\): the change of \(g\) as \(x\) increases
I hope that you can now see where the dominos come from. And good news, these are just derivatives. We can compute them one by one.
Let’s start with \(\frac{dg}{dx}\):
\[ g(x) = 4x - 3 \]
\[ \frac{dg}{dx} = 4 \]
If this is unclear, please reread the chapter on derivatives.
The last piece of the puzzle is \(\frac{df}{dg}\). This answers the question: how much does \(f\) change as \(g\) increases? The function \(f\) is defined as:
\[ f(u) = u^2 \]
It just returns the squared value of its input. When \(g\) becomes \(f\)’s input, we get:
\[ f(g) = g^2 \]
We can then compute \(\frac{df}{dg}\):
\[ \frac{df}{dg} = 2g \]
As \(\frac{d}{du}(u^2) = 2u\).
Putting all these pieces together with the chain rule:
\[ \frac{d}{dx} f(g(x)) = \frac{df}{dg} \times \frac{dg}{dx} = 2g \times 4 = 8g \]
We know that \(g = 4x - 3\), so we get:
\[ \frac{d}{dx} f(g(x)) = 8(4x - 3) \]
The derivative of \(h(x) = (4x - 3)^2\) is \(h'(x) = 8(4x - 3)\). This should match the derivative you found in the previous exercise.
Exercise 10.2 Use the chain rule to find the derivative of \(h(x) = (2x + 1)^2\).
- Identify \(f\) and \(g\) such that \(h(x) = f(g(x))\).
- Compute \(\frac{df}{dg}\) and \(\frac{dg}{dx}\).
- Apply the chain rule to find \(h'(x)\).
Exercise 10.3 Use the chain rule to find the derivative of \(h(x) = (x^2 + 3)^2\).
Hint: You can use the function \(g(x) = x^2 + 3\).
10.2 Final Thoughts
The chain rule is a way to compute the derivative of a composite function, a function of a function. It can be intimidating at first.
This is the last building block we need. The next chapter will use the chain rule to compute the derivative of the mean squared error with respect to the slope of a linear regression, and minimise it with gradient descent. Exciting.
10.3 Solutions
Solution 10.1. Exercise 10.1
Expanding \(h(x) = (4x - 3)^2\) using \((a + b)^2 = a^2 + 2ab + b^2\) with \(a = 4x\) and \(b = -3\):
\[\begin{aligned} h(x) &= (4x)^2 + 2 \times 4x \times (-3) + (-3)^2 \\ &= 16x^2 - 24x + 9 \end{aligned}\]
Taking the derivative term by term:
\[ h'(x) = 32x - 24 = 8(4x - 3) \]
Solution 10.2. Exercise 10.2
We can write \(h(x) = f(g(x))\) with:
\[ f(u) = u^2 \quad \text{and} \quad g(x) = 2x + 1 \]
Computing the derivatives:
\[ \frac{df}{dg} = 2g \quad \text{and} \quad \frac{dg}{dx} = 2 \]
Applying the chain rule:
\[ h'(x) = \frac{df}{dg} \times \frac{dg}{dx} = 2g \times 2 = 4g = 4(2x + 1) \]
Solution 10.3. Exercise 10.3
We can write \(h(x) = f(g(x))\) with:
\[ f(u) = u^2 \quad \text{and} \quad g(x) = x^2 + 3 \]
Computing the derivatives:
\[ \frac{df}{dg} = 2g \quad \text{and} \quad \frac{dg}{dx} = 2x \]
Applying the chain rule:
\[ h'(x) = \frac{df}{dg} \times \frac{dg}{dx} = 2g \times 2x = 4x(x^2 + 3) \]