Sparsity Comes from Score Centering

All of this is about the expected update. Whether a stored weight ends up changing also depends on the optimizer and on precision. Adam normalizes per coordinate, so a small but consistently signed gradient still moves a parameter by about one learning rate (Keeping the Off-Policy Gap Fixed). Mukherjee et al. (2025) count a bf16 weight as unchanged when it moves by at most $10^{-5}$, and note that at unlimited precision the updates would be dense. The bound says which coordinates get a small expected update; the optimizer and the precision decide whether that leaves the weight unchanged.

1. Output layer: a row can move only if the sampler emits its token

In the output layer the Fisher is an exact coin flip: row $c$ can move only if the sampler emits token $c$, and hardly at all once it is sure of it.

← ▶ → 5 / 5

Summed over a response, a row’s Fisher tracks how often the sampler emits its token: rows the task never uses get almost nothing.

$$ z_a=\sum_k W_{a,k}\,h_k\ \Rightarrow\ \kappa_{(c,k),a}=\mathbf 1[a=c]\,h_k,\qquad \bar\kappa^{\mu}_{(c,k)}=\smu(c)\,h_k, $$ $$ f_{(c,k)}=h_k^2\,\smu(c)\big(1-\smu(c)\big),\qquad \sum_{c,k}f_{(c,k)}=\|h\|^2\Big(1-\sum_c\smu(c)^2\Big), $$ $$ F^{\mu}_{\text{row }c}\le\max_s\|h(s)\|^2\cdot\mathbb{E}_{\smu}\big[\#\{t:\ y_t=c\}\big]. $$

The weight $W_{c,k}$ touches only token $c$’s logit, so for it the next token is a coin flip, $c$ or not $c$. Its Fisher is the coin’s variance times $h_k^2$: near zero when the sampler almost never considers $c$, and near zero when it is almost sure of it. With tea at 0.5, coffee at 0.4 and 0.1 spread over a hundred thousand other tokens, the total is about $1-0.25-0.16=0.59$, and the tea and coffee rows hold $0.25+0.24=0.49$ of it: 83%. The other 17% is spread over a hundred thousand rows, about a millionth each.

The two factors have different sources. $\smu(c)$ comes from the sampler and decides which rows matter: without centering the second moment would be $\smu(c)\,h_k^2$, concentrated on the same rows. Centering contributes the factor $1-\smu(c)$, which silences a token the sampler is sure of. Its main job is elsewhere: only a centered update is capped by this Fisher at all (section 3).

Along a response, $\mathbb{E}[\smu(c\mid s_t)]=\Pr(y_t=c)$, so a row’s Fisher is bounded by how often the sampler emits its token. A task emits a few thousand of the vocabulary’s hundred thousand tokens; the other rows barely move. The total is also the expected squared logit update of Logit Dynamics: averaged over the drawn token, its $1-2P_c+C(P)$ becomes $1-C(P)$, with $C(P)=\sum_aP_a^2$.

The whole output block, and a bound on concentration

$$ f_W=\big(\operatorname{diag}(\smu)-\smu\,\smu^\top\big)\otimes h\,h^\top . $$

The left factor is the covariance of a categorical distribution. Its off-diagonal entries are negative: raising one token crowds out another. Since $\smu(c)(1-\smu(c))\le\smu(c)$, for any set of tokens $S$:

$$ \frac{\sum_{c\notin S}\sum_k f_{(c,k)}}{\sum_c\sum_k f_{(c,k)}}\le\frac{\sum_{c\notin S}\smu(c)}{1-\sum_c\smu(c)^2}. $$

2. Hidden layers: few directions, not few coordinates

In hidden layers the Fisher sees only the few directions that tell the likely candidates apart.

← ▶ → 4 / 4

A parameter that moves h along that spread has Fisher; one that moves it across has none.

$$ z_a=w_a^\top h,\quad u_i=\frac{\partial h}{\partial\theta_i}\ \Rightarrow\ \kappa_{i,a}=w_a^\top u_i,\qquad \mathbb{E}_{\smu}\big[\kappa_{i,a}^2\big]=u_i^\top\big(C_\mu+\bar w_\mu\bar w_\mu^\top\big)u_i, $$ $$ f_i(s)=\operatorname{Var}_{a\sim\smu}\big[\kappa_{i,a}\big]=u_i^\top C_\mu\,u_i, $$ $$ C_\mu=\sum_a\smu(a)\,(w_a-\bar w_\mu)(w_a-\bar w_\mu)^\top,\qquad \operatorname{rank}C_\mu\le n-1. $$

Centering deletes one direction, the mean output vector $\bar w_\mu$, along which every likely logit rises together. What remains is the covariance of the candidates. If the sampler puts mass on $n$ tokens, their deviations from the mean sum to zero, so they span at most $n-1$ directions. In the extreme where $\smu$ sits on one token, every parameter’s Fisher is zero after centering, while the uncentered second moment is not. Short of that extreme, concentration alone already confines the second moment to the $n$ directions $w_a$; centering removes one more, the common direction, which may carry most of the energy. How much of the Fisher’s smallness that accounts for is open (section 6).

This is low dimension in function space, not coordinate sparsity. That would need most $u_i$ to be nearly orthogonal to these directions, which depends on the architecture and on pretraining, and the directions change from prefix to prefix. The evidence is empirical. In RL on Qwen3-1.7B-Base, Zhang, Mukherjee & Peng (2026) find that training only the top 0.1% of parameters by initial Fisher recovers 85.21% of the full-training gain on MATH-500, and 5% recovers 98.82%. The token Fisher under the model’s own probabilities is 0.16 of its value under a uniform distribution at the start, and 0.04 after 1,000 steps. These numbers show that the Fisher picks out the coordinates that matter; they do not separate the sampler’s concentration from centering. Sparse is also not low rank: the measured updates touch few coordinates yet are nearly full rank in every matrix (Mukherjee et al., 2025).

3. Without centering: the mean shift

Without centering, a mean shift adds up along the sequence and can push gradient into directions the Fisher cannot see.

← ▶ → 4 / 4

In the output layer, a demonstrated token the model gives 1% gets a push ten times its row’s on-policy cap.

$$ m_i(s)=\bar\kappa^{\mu}_i(s)-\bar\kappa^{p}_i(s),\qquad \mathbb{E}_{a\sim\smu}\big[(\kappa_{i,a}-\bar\kappa^{p}_i)^2\big]=\operatorname{Var}_{a\sim\smu}\big[\kappa_{i,a}\big]+m_i^2, $$ $$ \mathbb{E}_{\smu}\big[\mathrm{score}_i^{(u)}\,\mathrm{score}_i^{(t)}\big]=\mathbb{E}_{\smu}\big[\mathrm{score}_i^{(u)}\,m_i(s_t)\big]\ne0\quad(u\lt t). $$

Draw tokens from $\smu$ but subtract only the model’s mean, as the plain softmax score does, and each position averages to $m_i$ instead of zero. When the shift keeps its sign, the positions add up coherently: roughly $T\sigma^2+T^2m^2$ against $T\sigma^2$ with centering, an order-of-magnitude estimate rather than a theorem.

SFT is the extreme case, and a different one. Its sampler, the data’s next-token distribution, is unknown: each prefix shows a single token, so the score cannot be centered under it. Prefixes come from the data, and by the tower property the gradient is exactly the accumulated shift:

$$ \frac{\partial}{\partial\theta_i}\,\mathbb{E}_{\text{data}}\big[\log\pi_\theta(y\mid x)\big]=\mathbb{E}_{\text{data}}\Big[\sum_t m_i(s_t)\Big]. $$

Even with a known sampler, centering would delete the whole signal: with $A\equiv1$ the centered update is $\operatorname{Cov}_{\smu}(1,c_i)=0$. Imitation is the mean shift. So how sparse SFT is can only depend on how small the shift is, that is, on how far the data is from the model. The measurements agree: SFT on rejection-sampled, in-distribution data leaves 91% of parameters unchanged, and SFT on out-of-distribution data leaves 7% (Mukherjee et al., 2025). In the output layer the demonstrated token’s row receives $(1-\pmodel(y^\star))\,h$, nearly full size when the model gives $y^\star$ almost no probability, while the on-policy Fisher of that row is proportional to $\pmodel(y^\star)$. Data that leaves the model’s distribution touches parameters the model itself never touches.

The same cost, booked as a χ² signal

Let $D$ be the data distribution and $w=D/\pi_\theta$. Because the score has zero mean under $\pi_\theta$, any constant $b$ can be subtracted:

$$ G^{\text{SFT}}_i=\mathbb{E}_{\pi}\big[w\cdot\mathrm{score}_i\big]=\mathbb{E}_{\pi}\big[(w-b)\,\mathrm{score}_i\big],\qquad |G^{\text{SFT}}_i|\le\sqrt{\mathbb{E}_{\pi}\big[(w-b)^2\big]}\,\sqrt{F_i}, $$ $$ \mathbb{E}_{\pi}\big[(w-b)^2\big]=\chi^2(D\,\|\,\pi_\theta)+(1-b)^2\ \ge\ \chi^2(D\,\|\,\pi_\theta). $$

The weight is a product $w=\prod_t\rho_t$ with $\rho_t=q(y_t\mid s_t)/\pmodel(y_t\mid s_t)$, where $q$ is the data’s next-token distribution, and $1+\chi^2(D\|\pi_\theta)=\mathbb{E}_D[w]$. Conditioning on the last position gives $\mathbb{E}_D[\rho_T\mid s_T]=1+\chi^2\big(q(\cdot\mid s_T)\,\|\,\pmodel(\cdot\mid s_T)\big)$. If every position has $\chi^2(q\|p)\ge c$, backward induction over positions, with every ratio positive, gives

$$ 1+\chi^2(D\,\|\,\pi_\theta)\ \ge\ (1+c)^T . $$

The bound still holds, but its signal scale grows at least exponentially with distance from the model. It is the same $(1+\chi^2)^T$ form as the variance of sequence-level importance sampling in rl-collapse Part 2, and the $\chi^2$ that makes per-token importance sampling noisy in Around the Sampler. Booked directly under the data, the cost moves into the empirical Fisher $\mathbb{E}_D[\mathrm{score}_i^2]$, which a small $F_i$ does not bound.

4. Four cases side by side

The bound is a property of the estimator, not of RL.

Case	Samples from	Score centered under	Fisher bound
On-policy RL, on-policy distillation	the model	the model, for free	holds, with the model’s Fisher
Off-policy with score centering	a known sampler	the sampler	holds, with the sampler’s Fisher
SFT	the data, an unknown sampler	the model only; centering is impossible	holds formally; the signal scale is $\chi^2$
Value-based TD learning	a replay buffer	nothing	does not apply

TD weights a raw logit gradient, $\mathbb{E}[\delta\cdot\kappa_{i,a}]$, which keeps the common component: for a Q-value a uniform shift is the value level, not a redundancy. So the bound does not apply, which is not evidence that TD updates are dense. Two observations fit the table. DPO on a fixed preference set has an unknown sampler too, and it behaves like SFT: sparse on in-distribution data, dense on out-of-distribution data (Mukherjee et al., 2025). What separates the cases is the shift, not RL versus SFT. And SFT on expert traces made near-policy by MCMC sampling generalizes and forgets like on-policy methods (Karan, Chen & Du, 2026), as a small $\chi^2(D\,\|\,\pi_\theta)$ would suggest; that paper does not measure sparsity.

5. Sparsity is not a defect

Under training–inference mismatch, centering is what keeps numerical noise out of the update.

When the sampler is an inference engine, $\smu\approx\pmodel$, and the shift $m_i$ comes only from numerical differences between two implementations. It carries no reward information, yet it injects gradient into directions with near-zero on-policy Fisher and accumulates along the sequence. In the output layer the shift for row $c$ is $(\smu(c)-\pmodel(c))\,h$. Where the engine draws a token more often than the model would, $\smu(c)>\pmodel(c)$, that row is pushed although its on-policy Fisher is near zero. This is consistent with the mismatch doing most of its damage on rare tokens (When Speed Kills Stability).

Centering removes the shift exactly and keeps the update in the directions spanned by $C_\mu\approx C_p$. The engine’s full distribution is rarely available, but per-token importance weights do the same job with only the drawn token’s two probabilities: with $\rho_t=\pmodel(y_t\mid s_t)/\smu(y_t\mid s_t)$, $\mathbb{E}_{\smu}\big[\rho_t(\kappa_{i,y_t}-\bar\kappa^{p}_i)\big]=\sum_a\pmodel(a)\big(\kappa_{i,a}-\bar\kappa^{p}_i\big)=0$ at every prefix. Clipping a token’s ratio breaks this. Trust Region Masking never clips a token, so it leaves the weights that do the centering intact; it only drops whole sequences that left the trust region. Sparsity here means not moving the parameters that only noise would push.

6. Scope and open questions

• What centering needs. The sampler’s full next-token distribution at every prefix: engine log-probabilities or a stale model. Human demonstrations have no such distribution, which is why SFT cannot be centered. Per-token importance weights need only the drawn token’s probability under both (section 5). With top-k or top-p, the sampler is the truncated, renormalized distribution. Estimating $\bar\kappa^{\mu}$ from a batch makes the cancellation approximate.
• What it changes. Off-policy, centering drops the pull toward the sampler; the coefficient $\beta$ in Around the Sampler adds it back on purpose.
• Two mechanisms, not yet separated. A concentrated sampler and candidates that react alike both lower the Fisher. Two clean tests: permute which tokens get which probabilities at fixed entropy, and measure per coordinate the share centering removes, $\rho_i=(\bar\kappa^{p}_i)^2/\sum_a\pmodel(a)\,\kappa_{i,a}^2$. If low-Fisher coordinates have $\rho_i\approx1$, the sparsity comes mostly from the common component, as an untested heuristic suggests: likely tokens have large logits, so $h$ may lean along $\bar w_\mu$.
• One step. The Fisher itself changes during training, which is where Part 3 starts.

Next. Part 3 turns the bound around: how far can one update move a response the model almost never produces?

References

• Y. Zhang, S. Mukherjee, H. Peng. Understanding LLM Parameter Update Sparsity through the Lens of Fisher. arXiv, 2026.
• S. Mukherjee, L. Yuan, D. Hakkani-Tür, H. Peng. Reinforcement Learning Finetunes Small Subnetworks in Large Language Models. NeurIPS 2025.
• A. Karan, S. Chen, Y. Du. Finetuning with Sampling: SFT Learns Better Than You Think. arXiv, 2026.
• Y. Li. Logit Dynamics in Softmax Policy Gradient Methods. arXiv, 2025.
• J. Liu, Y. Li, et al. When Speed Kills Stability: Demystifying RL Collapse from the Training-Inference Mismatch. Blog, 2025.
• Y. Li, J. Liu, et al. Trust Region Masking for Long-Horizon LLM Reinforcement Learning. ICML 2026.