30  Appendix: Deriving the Cross-Entropy Gradient

This appendix derives the gradient of the cross-entropy loss with respect to the weight vector \(\mathbf{w}\). It complements the fitting a logistic regression chapter.

As a reminder, the cross-entropy loss for a single observation \(i\) is defined as:

\[ \text{CE}_i = -\left[ y_i \cdot \ln(p_i) + (1 - y_i) \cdot \ln(1 - p_i) \right] \]

and for a dataset of \(n\) observations:

\[ \text{CE} = -\frac{1}{n} \sum_{i=1}^{n} \left[ y_i \ln(p_i) + (1 - y_i) \ln(1 - p_i) \right] \]

The goal is to show that:

\[ \nabla \text{CE}(\mathbf{w}) = \frac{1}{n} \mathbf{X}^T (\hat{\mathbf{y}} - \mathbf{y}) \]

Setting Up the Chain Rule

Just like the original MSE formulation, the cross-entropy formula does not contain the weight vector \(\mathbf{w}\) directly:

\[ \text{CE} = -\frac{1}{n} \sum_{i=1}^{n} \left[ y_i \ln(p_i) + (1 - y_i) \ln(1 - p_i) \right] \]

To compute the gradient with regards to \(\mathbf{w}\), we need to substitute. Let’s call \(z_i\) the linear part, before the sigmoid:

\[ z_i = \mathbf{x}_i^T \mathbf{w} \]

\[ p_i = \sigma(z_i) = \frac{1}{1 + e^{-z_i}} \]

Substituting this directly into the cross-entropy formula would look brutal. A more humane way to deal with this problem is to use the chain rule.

As with the MSE derivation, we can focus on a single observation \(i\) (the \(\frac{1}{n}\) and the summation are constants with regards to \(\mathbf{w}\)).

For observation \(i\), the cross-entropy loss is:

\[ \text{CE}_i = -\left[ y_i \ln(p_i) + (1 - y_i) \ln(1 - p_i) \right] \]

We want to find \(\frac{\partial \text{CE}_i}{\partial \mathbf{w}}\). Using the chain rule:

\[ \frac{\partial \text{CE}_i}{\partial \mathbf{w}} = \frac{\partial \text{CE}_i}{\partial p_i} \times \frac{\partial p_i}{\partial z_i} \times \frac{\partial z_i}{\partial \mathbf{w}} \]

As a reminder, \(z_i = \mathbf{x}_i^T \mathbf{w}\). Think of this as three dominos falling:

\[ \mathbf{w} \text{ changes} \rightarrow z_i \text{ changes} \rightarrow p_i \text{ changes} \rightarrow \text{CE}_i \text{ changes} \]

Let’s compute each piece.

Piece 1: \(\frac{\partial \text{CE}_i}{\partial p_i}\)

Starting from:

\[ \text{CE}_i = -\left[ y_i \ln(p_i) + (1 - y_i) \ln(1 - p_i) \right] \]

To differentiate this with regards to \(p_i\), we need the derivative of \(\ln(p)\) and the derivative of \(\ln(1 - p)\) with respect to \(p\).

The first is straightforward: \(\frac{d}{dp} \ln(p) = \frac{1}{p}\).

For the second, we can use the chain rule. Let \(u = 1 - p\), so \(\ln(1 - p) = \ln(u)\):

\[ \frac{d}{dp} \ln(1 - p) = \frac{d}{du} \ln(u) \times \frac{du}{dp} = \frac{1}{u} \times (-1) = \frac{-1}{1 - p} \]

With these two results:

\[ \frac{\partial \text{CE}_i}{\partial p_i} = -\left[ \frac{y_i}{p_i} - \frac{1 - y_i}{1 - p_i} \right] \]

Combining these two fractions over a common denominator:

\[ \frac{\partial \text{CE}_i}{\partial p_i} = -\left[ \frac{y_i(1 - p_i) - (1 - y_i)p_i}{p_i(1 - p_i)} \right] = -\left[ \frac{y_i - p_i}{p_i(1 - p_i)} \right] = \frac{p_i - y_i}{p_i(1 - p_i)} \]

Piece 2: \(\frac{\partial p_i}{\partial z_i}\)

This piece answers the question: how does the predicted probability \(p_i = \sigma(z_i)\) change when \(z_i\) changes?

It turns out that the sigmoid has a remarkably elegant derivative:

\[ \frac{d\sigma}{dz} = \sigma(z) \times (1 - \sigma(z)) = p_i(1 - p_i) \]

Starting from \(\sigma(z) = \frac{1}{1 + e^{-z}}\), we can rewrite it as:

\[ \sigma(z) = (1 + e^{-z})^{-1} \]

Using the chain rule with \(f(u) = u^{-1}\) and \(g(z) = 1 + e^{-z}\):

\[ \frac{d\sigma}{dz} = \frac{df}{dg} \times \frac{dg}{dz} \]

With:

\[ \frac{df}{dg} = -g^{-2} = -(1 + e^{-z})^{-2} \]

\[ \frac{dg}{dz} = -e^{-z} \]

Substituting both back:

\[ \frac{d\sigma}{dz} = -(1 + e^{-z})^{-2} \times (-e^{-z}) = \frac{e^{-z}}{(1 + e^{-z})^2} \]

We can rewrite this as:

\[ \frac{e^{-z}}{(1 + e^{-z})^2} = \frac{1}{1 + e^{-z}} \times \frac{e^{-z}}{1 + e^{-z}} = \sigma(z) \times \frac{e^{-z}}{1 + e^{-z}} \]

Since \(\frac{e^{-z}}{1 + e^{-z}} = 1 - \frac{1}{1 + e^{-z}} = 1 - \sigma(z)\), we get:

\[ \frac{d\sigma}{dz} = \sigma(z)(1 - \sigma(z)) \]

Piece 3: \(\frac{\partial z_i}{\partial \mathbf{w}}\)

Since \(z_i = \mathbf{x}_i^T \mathbf{w}\), this is simply:

\[ \frac{\partial z_i}{\partial \mathbf{w}} = \mathbf{x}_i \]

This is the same result as in the multiple linear regression chapter: when the weights change, \(z_i\) changes proportionally to \(\mathbf{x}_i\).

Putting It Together

Multiplying the three pieces:

\[\begin{aligned} \frac{\partial \text{CE}_i}{\partial \mathbf{w}} &= \frac{\partial \text{CE}_i}{\partial p_i} \times \frac{\partial p_i}{\partial z_i} \times \frac{\partial z_i}{\partial \mathbf{w}} \\ &= \frac{p_i - y_i}{p_i(1 - p_i)} \times p_i(1 - p_i) \times \mathbf{x}_i \\ &= (p_i - y_i) \times \mathbf{x}_i \end{aligned}\]

Notice how \(p_i(1 - p_i)\) cancels out. The gradient for a single observation is simply the error \((p_i - y_i)\) multiplied by the input \(\mathbf{x}_i\).

Summing over all observations and dividing by \(n\):

\[ \nabla \text{CE}(\mathbf{w}) = \frac{1}{n} \sum_{i=1}^{n} (p_i - y_i) \mathbf{x}_i = \frac{1}{n} \mathbf{X}^T (\hat{\mathbf{y}} - \mathbf{y}) \]

This is the gradient used in the fitting chapter.