The Gradient Is a Covariance

The figures step through like slides. Use → or ▶, and step back with ←.

1. Sparsity needs its own explanation

A small update that carries few bits can still touch every coordinate.

RL fine-tuning changes only 5–30% of a model’s parameters, across seven algorithms and ten models (Mukherjee et al., 2025). On-policy distillation and SFT on near-policy data behave the same way (Zhang, Mukherjee & Peng, 2026). Two common explanations are that RL only reweights answers the model can already produce, and that a scalar advantage with $B$ possible values carries at most $\log_2 B$ bits per episode (Information Bandwidth). The first explains why the update is small, the second why it carries little; a hundred million parameters each moving a little would satisfy both. The question is which coordinates move, and why the same few can be found before any reward is seen.

2. The gradient is a covariance, the Fisher a variance

On the model’s own samples the score averages to zero, so the expected gradient is a covariance and the Fisher is a variance.

Probabilities sum to one, so whatever a parameter does, the responses it raises are paid for by others, and weighted by probability the changes cancel. With the score’s mean at zero, the policy gradient $\mathbb{E}[A\cdot\mathrm{score}_i]$ is a covariance and the Fisher is a variance; Cauchy–Schwarz gives the bound. A baseline that depends only on the prompt changes nothing in $G_i$, since $\operatorname{Cov}(A-b,\mathrm{score}_i)=\operatorname{Cov}(A,\mathrm{score}_i)$. It changes only $\operatorname{std}(A)$ and the noise of the estimate, and the Optimal Token Baseline, set per token, is the choice that minimizes that noise.

The Fisher says where learning can happen; the reward decides where it does. After “She drank a cup of”, the output rows for tea and coffee have a large Fisher, yet if the reward does not care which one is chosen, their gradient is zero. Measured before RL training, the top 1% of coordinates by Fisher hold 47.40% of the top 1% by gradient magnitude, against 1% at random (Zhang, Mukherjee & Peng, 2026).

3. One position: the Fisher is a spread of sensitivities

At one prefix, a parameter’s token Fisher is the variance of how strongly it moves each candidate’s logit.

The softmax subtracts a mean: the token score is how strongly $\theta_i$ moves the drawn token’s logit, minus how strongly it moves the average candidate. Without that subtraction the second moment would be $f_i(s)+(\bar\kappa^{p}_i)^2$. The removed part is the common component, which raises every logit together and changes no probability. The pairwise form says the same thing: only differences between likely candidates count. So a parameter has a small token Fisher exactly when it acts about equally on the candidates the model finds likely, either because the model is confident or because those candidates react alike.

4. Along the response: the Fisher adds up

Across positions the cross terms average to zero, so the Fisher is a sum of token Fishers.

Here $\mathrm{score}_i^{(t)}=\kappa_{i,y_t}-\bar\kappa^{p}_i(s_t)$ is the token score at position $t$. Given the prefix $s_t$, the next token is drawn from $\pmodel(\cdot\mid s_t)$, so its score averages to zero. For $u\lt t$ the earlier token is part of that prefix: its score is already fixed and comes out of the inner expectation. Summing over positions makes the Fisher grow with length and treats every parameter alike, so length sets the overall scale and the token Fishers decide which coordinates are large.

5. What this needs: a centered score

Everything above holds for any sampler whose next-token distribution is known, once the score is centered under it.

Off-policy, the token was drawn by an inference engine or a stale model, and under that sampler the softmax score averages to $\bar\kappa^{\mu}_i-\bar\kappa^{p}_i$. Subtracting this average at every prefix is score centering (Marek & Ryabinin, 2026). Sections 2–4 then go through word for word with $\smu$ in place of $\pmodel$; on-policy the average is already zero, so centering comes free. The centered update is the gradient of the covariance loss in Around the Sampler: the drawn token is raised and every token is lowered by how often the sampler draws it. Centering the score needs the sampler’s full next-token distribution at every prefix, which stale models provide, and so do inference engines that return full log-probabilities. Subtracting from the signal its average under the sampler at each prefix, a value baseline that can be fitted on the samples, has the same effect in expectation and needs no probabilities. SFT gains nothing from either: its signal is constant, so centering removes its whole gradient (Part 2).

Next. Part 2 puts this to work. In the output layer the Fisher is an exact coin flip, concentrated on the tokens the sampler emits; in hidden layers it is low-dimensional; and without centering a mean shift can push gradient where the Fisher cannot see.

References

• Y. Zhang, S. Mukherjee, H. Peng. Understanding LLM Parameter Update Sparsity through the Lens of Fisher. arXiv, 2026.
• S. Mukherjee, L. Yuan, D. Hakkani-Tür, H. Peng. Reinforcement Learning Finetunes Small Subnetworks in Large Language Models. NeurIPS 2025.
• M. Marek, M. Ryabinin. Score Centering Stabilizes Off-policy Reinforcement Learning. arXiv, 2026.
• Y. Li, J. Xu, et al. The Optimal Token Baseline: Variance Reduction for Long-Horizon LLM-RL. ICML 2026.
• Y. Li. Information Bandwidth in Reinforcement Learning. Blog, 2025.

Citation

@online{li-2026-gradient-is-covariance,
  title  = {The Gradient Is a Covariance},
  author = {Li, Yingru},
  year   = {2026},
  month  = oct,
  url    = {https://richardli.xyz/post/gradient-is-covariance/}
}