Fisher Information | Yingru Li

What Policy Gradient Can Reach

Thu, 15 Oct 2026 00:20:00 +0000

Why Post-Training Is Sparse · Part 3 of 3. Uses the covariance form of Part 1 and the Fisher of Part 2.

Reach follows resemblance. One update moves the log-odds of a response against another by at most std(A) · √(vᵀFv), with v their score difference and F the sampler’s Fisher.
Never drawn is not out of reach. A response that resembles nothing the sampler produces barely moves; one never drawn but similar to rewarded samples, like matcha after tea, moves with them.
Growth is proportional to probability. With one logit per response, getting from probability ε to one half takes time about 1/ε; a demonstration needs only about log(1/ε).
One step of one estimator. Shared structure, a changing Fisher, and a centered estimator with a different sampler all widen what is reachable.

1. A question that can be answered

Ask how much one update moves a response’s log-odds against another.

$$ \frac{d}{d\eta}\log\pi_{\theta+\eta G}(y\mid x)\Big|_{\eta=0}=\mathrm{score}(y)^\top G, $$ $$ \frac{d}{d\eta}\Big[\log\pi(y\mid x)-\log\pi(y'\mid x)\Big]_{\eta=0}=v^\top G, $$ $$ v=\mathrm{score}(y)-\mathrm{score}(y'). $$

“Can RL learn something new?” is too vague to prove anything about. A version that can be proved: for a response $y$ that the model rarely produces but the reward likes, how much can one update change its probability? The normalizer moves many responses at once, so the log-odds against a reference $y'$ isolates learning $y$ itself. Here $G=\mathbb{E}_{\smu}[A\cdot c]$ is the expected update of Part 1, and $\mathrm{score}(y)$ is the model’s score of the target. In the running example: after “She drank a cup of”, can matcha rise if the sampler never says it?

2. A response moves as far as it resembles the sampler’s outputs

One update moves a response only as far as its score resembles the scores of what the sampler draws.

5 / 5

A response off every direction the cloud covers barely moves, whatever its reward.

$$ \mathrm{score}(y)^\top G=\mathbb{E}_{y''\sim\smu}\big[A(y'')\,K(y,y'')\big], $$ $$ K(y,y'')=\mathrm{score}(y)^\top c(y''), $$ $$ \big|v^\top G\big|\le\operatorname{std}_{\smu}(A)\sqrt{v^\top F^{\mu}v}, $$ $$ v^\top F^{\mu}v=\mathbb{E}_{y''\sim\smu}\Big[\big(v^\top c(y'')\big)^2\Big]. $$

The similarity $K(y,y'')$ says how much raising a sampled $y''$ raises $y$ along the way. Matcha is never sampled, but if its score shares directions with tea’s, rewarding tea lifts matcha too. A response whose score is orthogonal to everything the sampler draws has $v^\top F^{\mu}v\approx0$ and does not move to first order, however high its reward. With $v$ a coordinate vector this is the bound of Part 1, now in any direction.

Proof, and the preconditioned version

By Part 1, $\mathbb{E}_{\smu}[c]=0$, so $\mathbb{E}_{\smu}[v^\top c]=0$ and the mean of $A$ can be subtracted:

$$ v^\top G=\mathbb{E}_{\smu}\big[(A-\mathbb{E}_{\smu}A)\,v^\top c\big] $$ $$ \le\sqrt{\operatorname{Var}_{\smu}(A)}\,\sqrt{\mathbb{E}_{\smu}\big[(v^\top c)^2\big]}=\operatorname{std}_{\smu}(A)\sqrt{v^\top F^{\mu}v}. $$

If the step is $\Delta\theta=\eta\,DG$ with a diagonal preconditioner $D$, as in one Adam step, replace $v$ with $Dv$: $|v^\top DG|\le\operatorname{std}_{\smu}(A)\sqrt{v^\top DF^{\mu}D\,v}$.

3. One logit per response: growth in proportion to probability

With one logit per response, a response grows at a rate proportional to its own probability.

3 / 3

A demonstration’s push does not shrink with the probability: a few units from either start.

Two responses, one logit each, reward gap 1.

$$ \frac{\partial\,\mathbb{E}_\pi[r]}{\partial z_y}=\pi(y)\big(r(y)-\mathbb{E}_\pi[r]\big)=\pi(y)\,\Delta(y), $$ $$ \frac{d\log\pi(y)}{d\eta}=\pi(y)\,\Delta(y)-\sum_{y''}\pi(y'')^2\,\Delta(y''), $$ $$ \frac{d}{d\eta}\big[\log\pi(y)-\log\pi(y')\big]=\pi(y)\,\Delta(y)-\pi(y')\,\Delta(y'). $$

The second term comes from the normalizer and is the same for every response: it raises or lowers the whole tail without changing ratios inside it. The log-odds keep only the first term, which is small for any response the model rarely produces. The kernel bound agrees: with $F=\operatorname{diag}(\pi)-\pi\pi^\top$ and $v=e_y-e_{y'}$, $v^\top Fv=\pi(y)+\pi(y')-(\pi(y)-\pi(y'))^2\le\pi(y)+\pi(y')$. Ignoring the normalizer, $d\pi/dt\approx\Delta\,\pi^2$, so $1/\pi$ falls at rate $\Delta$ and getting from $\varepsilon$ to one half takes about $(1/\varepsilon-2)/\Delta$. This matches the theory of softmax policy gradient, whose rate depends on how much probability the optimal action keeps (Mei et al., 2020), and which can take exponential time in the worst case (Li et al., 2021).

Drag the sliders. Each tenfold drop in the starting probability costs policy gradient about ten times more time, and the demonstration a constant.

The two-response case, solved exactly

With two responses, rewards $\Delta$ and 0, and log-odds $\ell=\log\frac{\pi}{1-\pi}$ starting at $\ell_0=\log\frac{\varepsilon}{1-\varepsilon}$:

$$ \text{policy gradient: }\ \frac{d\ell}{dt}=2\Delta\,\pi(1-\pi), $$ $$ t_{1/2}=\frac{e^{-\ell_0}-2\ell_0-e^{\ell_0}}{2\Delta}\approx\frac{1}{2\varepsilon\Delta}, $$ $$ \text{demonstration: }\ \frac{d\ell}{dt}=2(1-\pi), $$ $$ t_{1/2}=\frac{1-\ell_0-e^{\ell_0}}{2}\approx\tfrac12\log\frac1\varepsilon . $$

From $\varepsilon=0.01$ with $\Delta=1$ these give 54.1 and 2.8; from $\varepsilon=0.001$, 506 and 4.0.

4. A demonstration injects probability directly

A demonstration pushes a response at a rate that does not shrink with its probability.

$$ G=\mathrm{score}(y^\star)\ \Rightarrow\ \frac{d}{d\eta}\log\pi(y^\star)\Big|_{\eta=0}=\big\|\mathrm{score}(y^\star)\big\|^2, $$ $$ \big\|e_{y^\star}-\pi\big\|^2=\big(1-\pi(y^\star)\big)^2+\sum_{y\ne y^\star}\pi(y)^2 . $$

No similarity to sampled outputs enters, and the push stays near one however unlikely $y^\star$ is. The tabular expression is the squared logit update of Logit Dynamics for a chosen action, $1-2P_c+C(P)$; in the output layer, row $y^\star$ receives $(1-\pmodel(y^\star))\,h$ (Part 2). SFT can inject mass into a response the sampler never produces. Policy gradient can only amplify what it already reaches through similarity: $1/\varepsilon$ against $\log(1/\varepsilon)$ in the figure.

5. What the bound does not say

The bound limits one first-order step of one estimator. It does not say RL can only sharpen.

Shared structure. In a network the similarity $K$ is not diagonal. A response never sampled is lifted if it shares structure with rewarded samples, so new solutions recombined from existing pieces are within reach.
Many steps. The Fisher moves. In Zhang, Mukherjee & Peng (2026), the initial high-Fisher set keeps only 70–78% of its coordinates through 1,000 steps, and the mean Fisher of the initial low-Fisher set rises by 15–17%. Directions that were invisible can open; a multi-step bound would need control over that drift, and none exists yet.
First order. Large steps and strong nonlinearity add higher-order terms.
Which estimator, and which sampler. With full importance weighting, the expected update is the on-policy gradient, bounded by the model’s own Fisher whatever the sampler, so exploring alone does not widen a single step. With the score centered under the sampler, the update is $\operatorname{Cov}_{\smu}(A,c)$, bounded by $F^{\mu}$: a different sampler (a higher temperature, an exploration bonus, or posterior sampling as in Exploration at Scale) changes which directions are visible. Value-based TD learning is not score-weighted at all, so the bound does not apply to it.
Sparse is not little. A new capability can live in a few parameters.

6. What to measure instead

Whether policy gradient can learn a behavior depends on score similarity, not on probability alone.

$$ v^\top F^{\mu}v=\mathbb{E}_{y''\sim\smu}\Big[\big(v^\top c(y'')\big)^2\Big], $$ $$ v=\mathrm{score}(y)-\mathrm{score}(y'). $$

The debate over whether RL extends a base model’s abilities usually looks at probability: at large k, the base model’s pass@k can be higher (Yue et al., 2025). The bound points to a different quantity, the overlap between the target behavior’s score and what the sampler produces. Among equally unlikely responses, those that share structure with high-reward samples can be amplified and isolated ones cannot. Policy gradient is similarity-driven amplification. Whether it yields new capabilities depends on how new behavior connects, through shared parameters, to what the sampler already generates, and on which sampler is used.

Back to the start. Part 1 showed that a score centered under the sampler makes the update a covariance; Part 2 showed what that covariance can touch. Choosing the sampler is the exploration problem, which Exploration at Scale treats on its own terms.

References

J. Mei, C. Xiao, C. Szepesvári, D. Schuurmans. On the Global Convergence Rates of Softmax Policy Gradient Methods. ICML 2020.
G. Li, Y. Wei, Y. Chi, Y. Chen. Softmax Policy Gradient Methods Can Take Exponential Time to Converge. COLT 2021.
Y. Yue, Z. Chen, R. Lu, et al. Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? 2025.
Y. Li. Logit Dynamics in Softmax Policy Gradient Methods. arXiv, 2025.
Y. Zhang, S. Mukherjee, H. Peng. Understanding LLM Parameter Update Sparsity through the Lens of Fisher. arXiv, 2026.
Y. Li, J. Xu, B. Wang, Z.-Q. Luo. Scalable Exploration via Ensemble++. NeurIPS 2025.

Citation

@online{li-2026-what-policy-gradient-can-reach,
 title = {What Policy Gradient Can Reach},
 author = {Li, Yingru},
 year = {2026},
 month = oct,
 url = {https://richardli.xyz/post/what-policy-gradient-can-reach/}
}

Sparsity Comes from Score Centering

Thu, 15 Oct 2026 00:10:00 +0000

Why Post-Training Is Sparse · Part 2 of 3. Builds on Part 1, where a centered score makes the expected update a covariance that the Fisher caps. Part 3 asks what policy gradient can still reach.

Centering is the condition, not on-policy data. The Fisher caps the update only if the score has zero mean under the sampler. On-policy that comes free; off-policy, centering or per-token importance weights provide it.
Necessary, not sufficient. The cap makes the update sparse only if the Fisher is concentrated on a few coordinates. The reward then picks which of those move.
The Fisher is concentrated. In the output layer the formula is exact: a row’s Fisher is large only for tokens the sampler emits. In hidden layers only a few directions remain; coordinate sparsity there is measured, not derived.
Without centering, a mean shift adds up along the sequence and escapes the cap. Under training–inference mismatch it is numerical noise.
SFT is a different case. Its sampler is unknown, so it cannot be centered: its whole gradient is the mean shift, and it stays sparse only when the data is close to the model.

All of this is about the expected update. Whether a stored weight ends up changing also depends on the optimizer and on precision. Adam normalizes per coordinate, so a small but consistently signed gradient still moves a parameter by about one learning rate (Keeping the Off-Policy Gap Fixed). Mukherjee et al. (2025) count a bf16 weight as unchanged when it moves by at most $10^{-5}$, and note that at unlimited precision the updates would be dense. The bound says which coordinates get a small expected update; the optimizer and the precision decide whether that leaves the weight unchanged.

1. Output layer: a row can move only if the sampler emits its token

In the output layer the Fisher is an exact coin flip: row $c$ can move only if the sampler emits token $c$, and hardly at all once it is sure of it.

5 / 5

Summed over a response, a row’s Fisher tracks how often the sampler emits its token: rows the task never uses get almost nothing.

$$ z_a=\sum_k W_{a,k}\,h_k\ \Rightarrow\ \kappa_{(c,k),a}=\mathbf 1[a=c]\,h_k, $$ $$ \bar\kappa^{\mu}_{(c,k)}=\smu(c)\,h_k,\qquad f_{(c,k)}=h_k^2\,\smu(c)\big(1-\smu(c)\big), $$ $$ \sum_{c,k}f_{(c,k)}=\|h\|^2\Big(1-\sum_c\smu(c)^2\Big), $$ $$ F^{\mu}_{\text{row }c}\le\max_s\|h(s)\|^2\cdot\mathbb{E}_{\smu}\big[\#\{t:\ y_t=c\}\big]. $$

The weight $W_{c,k}$ touches only token $c$’s logit, so for it the next token is a coin flip, $c$ or not $c$. Its Fisher is the coin’s variance times $h_k^2$: near zero when the sampler almost never considers $c$, and near zero when it is almost sure of it. With tea at 0.5, coffee at 0.4 and 0.1 spread over a hundred thousand other tokens, the total is about $1-0.25-0.16=0.59$, and the tea and coffee rows hold $0.25+0.24=0.49$ of it: 83%. The other 17% is spread over a hundred thousand rows, about a millionth each.

The two factors have different sources. $\smu(c)$ comes from the sampler and decides which rows matter: without centering the second moment would be $\smu(c)\,h_k^2$, concentrated on the same rows. Centering contributes the factor $1-\smu(c)$, which silences a token the sampler is sure of. Its main job is elsewhere: only a centered update is capped by this Fisher at all (section 3).

Along a response, $\mathbb{E}[\smu(c\mid s_t)]=\Pr(y_t=c)$, so a row’s Fisher is bounded by how often the sampler emits its token. A task emits a few thousand of the vocabulary’s hundred thousand tokens; the other rows barely move. The total is also the expected squared logit update of Logit Dynamics: averaged over the drawn token, its $1-2P_c+C(P)$ becomes $1-C(P)$, with $C(P)=\sum_aP_a^2$.

Drag the sliders. As the sampler grows sure of one token the total Fisher collapses, and what is left sits in the rows it still hesitates over.

The whole output block, and a bound on concentration

$$ f_W=\big(\operatorname{diag}(\smu)-\smu\,\smu^\top\big)\otimes h\,h^\top . $$

The left factor is the covariance of a categorical distribution. Its off-diagonal entries are negative: raising one token crowds out another. Since $\smu(c)(1-\smu(c))\le\smu(c)$, for any set of tokens $S$:

$$ \frac{\sum_{c\notin S}\sum_k f_{(c,k)}}{\sum_c\sum_k f_{(c,k)}}\le\frac{\sum_{c\notin S}\smu(c)}{1-\sum_c\smu(c)^2}. $$

2. Hidden layers: few directions, not few coordinates

In hidden layers the Fisher sees only the few directions that tell the likely candidates apart.

4 / 4

A parameter that moves h along that spread has Fisher; one that moves it across has none.

$$ z_a=w_a^\top h,\quad u_i=\frac{\partial h}{\partial\theta_i}\ \Rightarrow\ \kappa_{i,a}=w_a^\top u_i, $$ $$ \mathbb{E}_{\smu}\big[\kappa_{i,a}^2\big]=u_i^\top\big(C_\mu+\bar w_\mu\bar w_\mu^\top\big)u_i, $$ $$ f_i(s)=\operatorname{Var}_{a\sim\smu}\big[\kappa_{i,a}\big]=u_i^\top C_\mu\,u_i, $$ $$ C_\mu=\sum_a\smu(a)\,(w_a-\bar w_\mu)(w_a-\bar w_\mu)^\top, $$ $$ \operatorname{rank}C_\mu\le n-1. $$

Centering deletes the rank-one term $\bar w_\mu\bar w_\mu^\top$: what a parameter does to the average likely logit, $\bar\kappa^{\mu}_i=\bar w_\mu^\top u_i$. Raising every likely logit alike moves no probability, so that part counts for nothing. What remains is the covariance of the candidates. If the sampler puts mass on $n$ tokens, their deviations from the mean sum to zero, so they span at most $n-1$ directions. In the extreme where $\smu$ sits on one token, every parameter’s Fisher is zero after centering, while the uncentered second moment is not. Short of that extreme, concentration alone already confines the second moment to the $n$ directions $w_a$; centering removes the common part on top, which may carry most of the energy. How much of the Fisher’s smallness that accounts for is open (section 6).

This is low dimension in function space, not coordinate sparsity. That would need most $u_i$ to be nearly orthogonal to these directions, which depends on the architecture and on pretraining, and the directions change from prefix to prefix. The evidence is empirical. In RL on Qwen3-1.7B-Base, Zhang, Mukherjee & Peng (2026) find that training only the top 0.1% of parameters by initial Fisher recovers 85.21% of the full-training gain on MATH-500, and 5% recovers 98.82%. The token Fisher under the model’s own probabilities is 0.16 of its value under a uniform distribution at the start, and 0.04 after 1,000 steps. These numbers show that the Fisher picks out the coordinates that matter. The authors also report that likely next tokens tend to have similar sensitivities, which is the common part centering removes; but the model-versus-uniform ratio changes concentration and token assignment together, so the two mechanisms are not yet separated (section 6). Sparse is also not low rank: the measured updates touch few coordinates yet are nearly full rank in every matrix (Mukherjee et al., 2025).

3. Without centering: the mean shift

Without centering, a mean shift adds up along the sequence and can push gradient into directions the Fisher cannot see.

4 / 4

In the output layer, a demonstrated token the model gives 1% gets a push ten times its row’s on-policy cap.

$$ m_i(s)=\bar\kappa^{\mu}_i(s)-\bar\kappa^{p}_i(s), $$ $$ \mathbb{E}_{a\sim\smu}\big[(\kappa_{i,a}-\bar\kappa^{p}_i)^2\big]=\operatorname{Var}_{a\sim\smu}\big[\kappa_{i,a}\big]+m_i^2, $$ $$ \mathbb{E}_{\smu}\big[\mathrm{score}_i^{(u)}\,\mathrm{score}_i^{(t)}\big] $$ $$ =\mathbb{E}_{\smu}\big[\mathrm{score}_i^{(u)}\,m_i(s_t)\big]\ne0\quad(u\lt t). $$

Draw tokens from $\smu$ but subtract only the model’s mean, as the plain softmax score does, and each position averages to $m_i$ instead of zero. When the shift keeps its sign, the positions add up coherently: roughly $T\sigma^2+T^2m^2$ against $T\sigma^2$ with centering, an order-of-magnitude estimate rather than a theorem.

SFT is the extreme case, and a different one. Its sampler, the data’s next-token distribution, is unknown: each prefix shows a single token, so the score cannot be centered under it. Prefixes come from the data, and by the tower property the gradient is exactly the accumulated shift:

$$ \frac{\partial}{\partial\theta_i}\,\mathbb{E}_{\text{data}}\big[\log\pi_\theta(y\mid x)\big]=\mathbb{E}_{\text{data}}\Big[\sum_t m_i(s_t)\Big]. $$

Even with a known sampler, centering would delete the whole signal: with $A\equiv1$ the centered update is $\operatorname{Cov}_{\smu}(1,c_i)=0$. Imitation is the mean shift. So how sparse SFT is can only depend on how small the shift is, that is, on how far the data is from the model. The measurements agree: SFT on rejection-sampled, in-distribution data leaves 91% of parameters unchanged, and SFT on out-of-distribution data leaves 7% (Mukherjee et al., 2025). In the output layer the demonstrated token’s row receives $(1-\pmodel(y^\star))\,h$, nearly full size when the model gives $y^\star$ almost no probability, while the on-policy Fisher of that row is proportional to $\pmodel(y^\star)$. Data that leaves the model’s distribution touches parameters the model itself never touches.

The same cost, booked as a χ² signal

Let $D$ be the data distribution and $w=D/\pi_\theta$. Because the score has zero mean under $\pi_\theta$, any constant $b$ can be subtracted:

$$ G^{\text{SFT}}_i=\mathbb{E}_{\pi}\big[w\cdot\mathrm{score}_i\big]=\mathbb{E}_{\pi}\big[(w-b)\,\mathrm{score}_i\big], $$ $$ |G^{\text{SFT}}_i|\le\sqrt{\mathbb{E}_{\pi}\big[(w-b)^2\big]}\,\sqrt{F_i}, $$ $$ \mathbb{E}_{\pi}\big[(w-b)^2\big]=\chi^2(D\,\|\,\pi_\theta)+(1-b)^2 $$ $$ \ge\ \chi^2(D\,\|\,\pi_\theta). $$

The weight is a product $w=\prod_t\rho_t$ with $\rho_t=q(y_t\mid s_t)/\pmodel(y_t\mid s_t)$, where $q$ is the data’s next-token distribution, and $1+\chi^2(D\|\pi_\theta)=\mathbb{E}_D[w]$. Conditioning on the last position gives $\mathbb{E}_D[\rho_T\mid s_T]=1+\chi^2\big(q(\cdot\mid s_T)\,\|\,\pmodel(\cdot\mid s_T)\big)$. If every position has $\chi^2(q\|p)\ge c$, backward induction over positions, with every ratio positive, gives

$$ 1+\chi^2(D\,\|\,\pi_\theta)\ \ge\ (1+c)^T . $$

The bound still holds, but its signal scale grows at least exponentially with distance from the model. It is the same $(1+\chi^2)^T$ form as the variance of sequence-level importance sampling in rl-collapse Part 2, and the $\chi^2$ that makes per-token importance sampling noisy in Around the Sampler. Booked directly under the data, the cost moves into the empirical Fisher $\mathbb{E}_D[\mathrm{score}_i^2]$, which a small $F_i$ does not bound.

4. Four cases side by side

The bound is a property of the estimator, not of RL.

Case	Samples from	Score centered under	Fisher bound
On-policy RL, on-policy distillation	the model	the model, for free	holds, with the model’s Fisher
Off-policy with score centering	a known sampler	the sampler	holds, with the sampler’s Fisher
SFT	the data, an unknown sampler	the model only; centering is impossible	holds formally; the signal scale is $\chi^2$
Value-based TD learning	a replay buffer	nothing	does not apply

TD weights a raw logit gradient, $\mathbb{E}[\delta\cdot\kappa_{i,a}]$, which keeps the common component: for a Q-value a uniform shift is the value level, not a redundancy. So the bound does not apply, which is not evidence that TD updates are dense. Two observations fit the table. DPO on a fixed preference set has an unknown sampler too, and it behaves like SFT: sparse on in-distribution data, dense on out-of-distribution data (Mukherjee et al., 2025). What separates the cases is the shift, not RL versus SFT. And SFT on expert traces made near-policy by MCMC sampling generalizes and forgets like on-policy methods (Karan, Chen & Du, 2026), as a small $\chi^2(D\,\|\,\pi_\theta)$ would suggest; that paper does not measure sparsity.

5. Sparsity is not a defect

Under training–inference mismatch, centering is what keeps numerical noise out of the update.

When the sampler is an inference engine, $\smu\approx\pmodel$, and the shift $m_i$ comes only from numerical differences between two implementations. It carries no reward information, yet it injects gradient into directions with near-zero on-policy Fisher and accumulates along the sequence. In the output layer the shift for row $c$ is $(\smu(c)-\pmodel(c))\,h$. Where the engine draws a token more often than the model would, $\smu(c)>\pmodel(c)$, that row is pushed although its on-policy Fisher is near zero. This is consistent with the mismatch doing most of its damage on rare tokens (When Speed Kills Stability).

Centering removes the shift exactly and keeps the update in the directions spanned by $C_\mu\approx C_p$. The engine’s full distribution is rarely available, but per-token importance weights do the same job with only the drawn token’s two probabilities: with $\rho_t=\pmodel(y_t\mid s_t)/\smu(y_t\mid s_t)$, $\mathbb{E}_{\smu}\big[\rho_t(\kappa_{i,y_t}-\bar\kappa^{p}_i)\big]=\sum_a\pmodel(a)\big(\kappa_{i,a}-\bar\kappa^{p}_i\big)=0$ at every prefix. Clipping a token’s ratio breaks this. Trust Region Masking never clips a token, so it leaves the weights that do the centering intact; it only drops whole sequences that left the trust region. Sparsity here means not moving the parameters that only noise would push.

6. Scope and open questions

What centering needs. The sampler’s full next-token distribution at every prefix: engine log-probabilities or a stale model. Human demonstrations have no such distribution, which is why SFT cannot be centered. Per-token importance weights need only the drawn token’s probability under both (section 5). With top-k or top-p, the sampler is the truncated, renormalized distribution. Estimating $\bar\kappa^{\mu}$ from a batch makes the cancellation approximate.
What it changes. Off-policy, centering drops the pull toward the sampler; the coefficient $\beta$ in Around the Sampler adds it back on purpose.
Two mechanisms, not yet separated. A concentrated sampler and candidates that react alike both lower the Fisher. Two clean tests: permute which tokens get which probabilities at fixed entropy, and measure per coordinate the share centering removes, $\rho_i=(\bar\kappa^{p}_i)^2/\sum_a\pmodel(a)\,\kappa_{i,a}^2$. If low-Fisher coordinates have $\rho_i\approx1$, the sparsity comes mostly from the common component, as an untested heuristic suggests: likely tokens have large logits, so $h$ may lean along $\bar w_\mu$.
One step. The Fisher itself changes during training, which is where Part 3 starts.

Next. Part 3 turns the bound around: how far can one update move a response the model almost never produces?

References

Y. Zhang, S. Mukherjee, H. Peng. Understanding LLM Parameter Update Sparsity through the Lens of Fisher. arXiv, 2026.
S. Mukherjee, L. Yuan, D. Hakkani-Tür, H. Peng. Reinforcement Learning Finetunes Small Subnetworks in Large Language Models. NeurIPS 2025.
A. Karan, S. Chen, Y. Du. Finetuning with Sampling: SFT Learns Better Than You Think. arXiv, 2026.
Y. Li. Logit Dynamics in Softmax Policy Gradient Methods. arXiv, 2025.
J. Liu, Y. Li, et al. When Speed Kills Stability: Demystifying RL Collapse from the Training-Inference Mismatch. Blog, 2025.
Y. Li, J. Liu, et al. Trust Region Masking for Long-Horizon LLM Reinforcement Learning. ICML 2026.

Citation

@online{li-2026-sparsity-score-centering,
 title = {Sparsity Comes from Score Centering},
 author = {Li, Yingru},
 year = {2026},
 month = oct,
 url = {https://richardli.xyz/post/sparsity-score-centering/}
}

The Gradient Is a Covariance

Thu, 15 Oct 2026 00:00:00 +0000

Why Post-Training Is Sparse · Part 1 of 3. Part 2 finds where the sparsity comes from. Part 3 asks what policy gradient can still learn.

RL, on-policy distillation and SFT on near-policy data change only a small fraction of parameters. Small and sparse are different claims.
On the sampler’s own samples the score has zero mean. So the expected gradient is a covariance, the Fisher is a variance, and the Fisher caps the gradient for any reward.
At one position the Fisher is the variance of logit sensitivities: whatever moves every candidate together counts for nothing.
None of this needs on-policy sampling. It needs a score centered under the sampling distribution.

The figures step through like slides. Use → or ▶, and step back with ←.

Symbol	Meaning
$\smu(a\mid s)$	the distribution that drew the next token after prefix $s$: the sampler
$\pmodel(a\mid s)$	the model’s next-token distribution; $\smu=\pmodel$ on-policy
$\kappa_{i,a}=\partial z_a/\partial\theta_i$	how strongly parameter $i$ moves the logit $z_a$ of token $a$
$A$	the learning signal: a reward or advantage, possibly times an importance ratio
$G_i,\ F_i$	the expected update and the diagonal Fisher of parameter $i$

1. Small is not sparse

Reweighting what the model already samples explains why the update is small, not why it lands on a few coordinates.

RL fine-tuning changes only 5–30% of a model’s parameters, across seven algorithms and ten models (Mukherjee et al., 2025). On-policy distillation and SFT on near-policy data behave the same way (Zhang, Mukherjee & Peng, 2026). Two common explanations are that RL only reweights answers the model can already produce, and that a scalar advantage carries at most $\log_2 B$ bits per episode (Information Bandwidth). The first says the update is small, the second that it carries little. Neither makes it sparse: a hundred million parameters each moving a little is small and dense, and can carry those few bits just as well. The question is which coordinates move, and why the same few can be found before any reward is seen.

2. The gradient is a covariance, the Fisher a variance

On the model’s own samples the score averages to zero, so the expected gradient is a covariance and the Fisher is a variance.

4 / 4

The line can never leave the wedge of perfect correlation, which is the cap |G| ≤ std(A) · √F: the same for both. A large Fisher allows an update; only the reward makes one.

$$ \mathrm{score}_i=\frac{\partial\log\pi_\theta(y\mid x)}{\partial\theta_i}, $$ $$ \mathbb{E}_{y\sim\pi_\theta}\big[\mathrm{score}_i\big]=\sum_y\partial_i\pi_\theta(y\mid x)=\partial_i 1=0, $$ $$ G_i=\mathbb{E}\big[A\cdot\mathrm{score}_i\big]=\operatorname{Cov}(A,\mathrm{score}_i), $$ $$ F_i=\mathbb{E}\big[\mathrm{score}_i^2\big]=\operatorname{Var}(\mathrm{score}_i), $$ $$ |G_i|\le\operatorname{std}(A)\sqrt{F_i}. $$

Probabilities sum to one, so whatever a parameter does, the responses it raises are paid for by others, and weighted by probability the changes cancel. With the score’s mean at zero, the policy gradient $\mathbb{E}[A\cdot\mathrm{score}_i]$ is a covariance and the Fisher is a variance; Cauchy–Schwarz gives the bound. A baseline that depends only on the prompt changes nothing in $G_i$, since $\operatorname{Cov}(A-b,\mathrm{score}_i)=\operatorname{Cov}(A,\mathrm{score}_i)$. It changes only $\operatorname{std}(A)$ and the noise of the estimate, and the Optimal Token Baseline is the choice that minimizes that noise.

A large Fisher is necessary for a large gradient, not sufficient. After “She drank a cup of”, the output rows for tea and coffee have a large Fisher; if the reward does not care which one is chosen, their gradient is zero. Measured before RL training, the top 1% of coordinates by Fisher hold 47.40% of the top 1% by gradient magnitude, against 1% at random (Zhang, Mukherjee & Peng, 2026). The Fisher says where learning can happen; the reward decides where it does.

Proof, and the four assumptions it uses

The response space is finite (finite vocabulary, bounded length), so every exchange of a derivative and a sum below is exact. Probabilities are positive, which softmax sampling at temperature 1 without top-k or top-p guarantees. The reward $r(x,y)$ and the prompt distribution do not depend on $\theta$.

$$ \mathbb{E}_{y\sim\pi_\theta}[\mathrm{score}_i]=\sum_y\pi_\theta\,\frac{\partial_i\pi_\theta}{\pi_\theta}=\partial_i\sum_y\pi_\theta=0, $$ $$ G_i=\partial_i\,\mathbb{E}_x\Big[\sum_y\pi_\theta\,r\Big] $$ $$ =\mathbb{E}_x\Big[\sum_y\pi_\theta\,\mathrm{score}_i\,r\Big]=\mathbb{E}[r\cdot\mathrm{score}_i]. $$

Subtracting $\mathbb{E}[r]\,\mathbb{E}[\mathrm{score}_i]=0$ turns the product into a covariance, and the same step turns $\mathbb{E}[\mathrm{score}_i^2]$ into a variance. Both hold per prompt, so $G_i=\mathbb{E}_x[\operatorname{Cov}(r,\mathrm{score}_i\mid x)]$, which is why a baseline $b(x)$ drops out.

The Fisher is also the expected curvature: $\partial_j\mathrm{score}_i=\partial_i\partial_j\pi_\theta/\pi_\theta-\mathrm{score}_i\,\mathrm{score}_j$, and the first term has mean $\sum_y\partial_i\partial_j\pi_\theta=0$, so $F_{ij}=-\mathbb{E}[\partial_i\partial_j\log\pi_\theta]$. It must be estimated on responses the model samples. On data labels the same formula gives the empirical Fisher, a different quantity (Part 2).

3. One position: the Fisher is a spread of sensitivities

At one prefix, a parameter’s token Fisher is the variance of how strongly it moves each candidate’s logit.

6 / 6

So do likely candidates that react alike: over thirty times smaller.

$$ \frac{\partial\log\pmodel(a\mid s)}{\partial\theta_i}=\kappa_{i,a}-\bar\kappa^{p}_i, $$ $$ \bar\kappa^{p}_i=\sum_a\pmodel(a\mid s)\,\kappa_{i,a}, $$ $$ f_i(s)=\operatorname{Var}_{a\sim\pmodel}\big[\kappa_{i,a}\big] $$ $$ =\tfrac12\sum_{a,b}\pmodel(a)\,\pmodel(b)\,\big(\kappa_{i,a}-\kappa_{i,b}\big)^2 . $$

The softmax subtracts a mean: the token score is how strongly $\theta_i$ moves the drawn token’s logit, minus how strongly it moves the average candidate. Without that subtraction the second moment would be $f_i(s)+(\bar\kappa^{p}_i)^2$. The removed part is the common component, which raises every logit together and changes no probability. The pairwise form says the same thing: only differences between likely candidates count. So a parameter has a small token Fisher exactly when it acts about equally on the candidates the model finds likely, either because the model is confident or because those candidates react alike.

4. Along the response: the Fisher adds up

Across positions the cross terms average to zero, so the Fisher is a sum of token Fishers.

5 / 5

Only the diagonal survives: the Fisher adds one token Fisher per position.

$$ \mathrm{score}_i=\sum_t\mathrm{score}_i^{(t)}, $$ $$ \mathbb{E}\big[\mathrm{score}_i^{(u)}\,\mathrm{score}_i^{(t)}\big] $$ $$ =\mathbb{E}\Big[\mathrm{score}_i^{(u)}\;\underbrace{\mathbb{E}\big[\mathrm{score}_i^{(t)}\mid s_t\big]}_{=0}\Big]=0\quad(u\lt t), $$ $$ F_i=\mathbb{E}\Big[\sum_t f_i(s_t)\Big]. $$

Given the prefix $s_t$, the next token is drawn from $\pmodel(\cdot\mid s_t)$, so its score averages to zero. For $u\lt t$ the earlier token is part of that prefix: its score is already fixed and comes out of the inner expectation. The cancellation holds in expectation, not sample by sample. It makes the Fisher grow with length but treats every parameter alike, so it sets the scale, not which coordinates are large.

5. What this needs: a centered score

None of this needs on-policy sampling. It needs the score centered under the distribution that drew the samples.

the sampler’s distribution at one prefix

the drawn token: raised

every token: lowered by the sampler’s probability

3 / 3

Averaged over what the sampler draws, it is zero: a constant reward moves nothing.

Score centering, public reference: Marek & Ryabinin, 2026.

$$ c_i^{(t)}=\kappa_{i,y_t}-\bar\kappa^{\mu}_i(s_t),\qquad \bar\kappa^{\mu}_i=\sum_a\smu(a\mid s)\,\kappa_{i,a}, $$ $$ G_i=\mathbb{E}_{\smu}\big[A\,c_i\big]=\operatorname{Cov}_{\smu}(A,c_i), $$ $$ |G_i|\le\operatorname{std}_{\smu}(A)\sqrt{F^{\mu}_i}, $$ $$ F^{\mu}_i=\mathbb{E}_{\smu}\Big[\sum_t\operatorname{Var}_{a\sim\smu}\big[\kappa_{i,a}(s_t)\big]\Big]. $$

Off-policy, the token was drawn by an inference engine or a stale model, and under that sampler the softmax score averages to $\bar\kappa^{\mu}_i-\bar\kappa^{p}_i$, not to zero. Subtracting it at every prefix is score centering. Sections 2–4 then go through word for word with $\smu$ in place of $\pmodel$, and on-policy the centering comes free. This is the gradient of the covariance loss in Around the Sampler: the drawn token is raised and every token is lowered by how often the sampler draws it. It needs the sampler’s full next-token distribution at every prefix, which engine log-probabilities and stale models provide and human demonstrations do not.

The theorem, proved

For any sampler $\smu$ whose full next-token distribution is known at every prefix:

Zero conditional mean, by construction: $\mathbb{E}_{a\sim\smu}\big[\kappa_{i,a}-\bar\kappa^{\mu}_i\big]=0$.
Cross terms vanish: for $u\lt t$, $c_i^{(u)}$ is fixed by $s_t$, so $\mathbb{E}_{\smu}\big[c_i^{(u)}c_i^{(t)}\big]=\mathbb{E}_{\smu}\big[c_i^{(u)}\,\mathbb{E}_{\smu}[c_i^{(t)}\mid s_t]\big]=0$.
The Fisher is a variance: by 2 only diagonal terms survive, and by 1 each is $\operatorname{Var}_{a\sim\smu}[\kappa_{i,a}(s_t)]$. By 1 and the tower property $\mathbb{E}_{\smu}[c_i]=0$, so the second moment is the variance.
The update is a covariance: $\operatorname{Cov}_{\smu}(A,c_i)=\mathbb{E}_{\smu}[A\,c_i]=G_i$, and Cauchy–Schwarz gives the bound.

On-policy, $c_i=\mathrm{score}_i$ and $G_i$ is the gradient. Off-policy, $G_i$ is the expected update of the centered estimator: the gradient of the covariance loss, not in general of the expected reward.

Next. Part 2 puts this to work. In the output layer the Fisher is an exact coin flip, concentrated on the tokens the sampler emits; in hidden layers it is low-dimensional; and without centering a mean shift can push gradient where the Fisher cannot see.

References

Y. Zhang, S. Mukherjee, H. Peng. Understanding LLM Parameter Update Sparsity through the Lens of Fisher. arXiv, 2026.
S. Mukherjee, L. Yuan, D. Hakkani-Tür, H. Peng. Reinforcement Learning Finetunes Small Subnetworks in Large Language Models. NeurIPS 2025.
M. Marek, M. Ryabinin. Score Centering Stabilizes Off-policy Reinforcement Learning. arXiv, 2026.
Y. Li, J. Xu, et al. The Optimal Token Baseline: Variance Reduction for Long-Horizon LLM-RL. ICML 2026.
Y. Li. Information Bandwidth in Reinforcement Learning. Blog, 2025.

Citation

@online{li-2026-gradient-is-covariance,
 title = {The Gradient Is a Covariance},
 author = {Li, Yingru},
 year = {2026},
 month = oct,
 url = {https://richardli.xyz/post/gradient-is-covariance/}
}