Sparsity Requires Score Centering

Starting point. Write $\smu$ for the sampler, $\pmodel$ for the model’s next-token distribution, $A$ for the learning signal (a reward or advantage), and $\kappa_{i,a}=\partial z_a/\partial\theta_i$ for how strongly parameter $i$ moves the logit of token $a$. For the token $y_t$ drawn after prefix $s_t$, the softmax score subtracts the average under the model, $\mathrm{score}_i^{(t)}=\kappa_{i,y_t}-\bar\kappa^{p}_i(s_t)$; centering subtracts the average under the sampler, $c_i^{(t)}=\kappa_{i,y_t}-\bar\kappa^{\mu}_i(s_t)$. Over a response of $T$ tokens, $c_i=\sum_{t=1}^{T}c_i^{(t)}$. With centering, the expected update is a covariance and the Fisher caps it (Part 1 has the derivation):

Sections 1 and 2 show where and why the Fisher concentrates; section 3 shows what goes wrong without centering.

This concerns the expected update; whether a stored weight changes also depends on Adam and on precision. Mukherjee et al. (2025) count a bf16 weight as unchanged within $10^{-5}$, and note that at unlimited precision the updates would be dense.

1. Output layer: an exact coin flip

In the output layer the Fisher is an exact coin flip: row $c$ can move only if the sampler emits token $c$, and hardly at all once it is sure of it.

The weight $W_{c,k}$ touches only token $c$’s logit, so for it the next token is a coin flip, $c$ or not $c$, and its Fisher is the coin’s variance times $h_k^2$: near zero when the sampler almost never considers $c$, or is almost sure of it. With tea at 0.5, coffee at 0.4 and 0.1 spread over a hundred thousand other tokens, the tea and coffee rows hold 83% of the total, and every other row about a millionth.

The factor $\smu(c)$ comes from the sampler and decides which rows matter; even the raw second moment $\smu(c)\,h_k^2$, with no mean subtracted, sits on the same rows. A diffuse sampler, such as high-temperature sampling, spreads $\smu(c)$, and with it the Fisher, however well the score is centered. Centering contributes $1-\smu(c)$, which silences a token the sampler is sure of; its main job is to make the cap hold at all (section 3).

Along a response, $\mathbb{E}[\smu(c\mid s_t)]=\Pr(y_t=c)$, so a row’s Fisher is bounded by how often the sampler emits its token. A task emits only a small part of the vocabulary, and the rows of the other tokens barely move.

2. Hidden layers: a few directions

In hidden layers the Fisher sees only the few directions that tell the likely candidates apart.

With $\bar w_\mu=\sum_a\smu(a)\,w_a$ the average output vector, centering deletes the rank-one term $\bar w_\mu\bar w_\mu^\top$: what a parameter does to the average likely logit, $\bar\kappa^{\mu}_i=\bar w_\mu^\top u_i$. Raising every likely logit alike moves no probability, so that common part counts for nothing, and what remains is the covariance of the candidates. If the sampler puts mass on $n$ tokens, their deviations from the mean span at most $n-1$ directions, one fewer than the $n$ of the raw second moment.

So the hidden-layer Fisher is small for two reasons: the sampler puts its mass on few tokens, and those tokens react alike, as Zhang, Mukherjee & Peng (2026) propose for likely next tokens. In their runs the token Fisher under the model’s own probabilities is 0.16 of its value under a uniform distribution at the start of RL and 0.04 after 1,000 steps: the model is a confident sampler, and RL makes it more so. How much each reason contributes is still open (section 6).

Few directions become few coordinates only if most $u_i$ are nearly orthogonal to these directions, which depends on the architecture and on pretraining, so here the evidence is empirical. In RL on Qwen3-1.7B-Base, training only the top 0.1% of parameters by initial Fisher recovers 85.21% of the full-training gain on MATH-500, and 5% recovers 98.82% (Zhang, Mukherjee & Peng, 2026). Sparsity and low rank are separate properties: the measured updates touch few coordinates and are nearly full rank in every matrix (Mukherjee et al., 2025). Part 4 shows that one expected step is low-rank in these directions, and what spreads the accumulated change over the rest.

3. Without centering: the mean shift

Without centering, a mean shift adds up along the sequence and can push gradient into directions the Fisher cannot see.

Draw tokens from $\smu$ but subtract only the model’s mean, as the plain softmax score does. At each prefix the expected update then splits in two: the covariance of the signal with the sensitivities, which the sampler’s Fisher caps, and the average signal times the mean shift, which escapes the cap. Centering removes the second part from either side. Subtracting $\bar\kappa^{\mu}_i$ from the score needs the sampler’s full distribution; subtracting the per-prefix value $V^{\mu}(s_t)=\mathbb{E}_{\smu}[A\mid s_t]$ from the signal needs only a baseline fitted on the samples, and gives the same expected update. Left in, the mean shift also adds up across positions: with per-position variance $\sigma^2$ and a shift $m$ of constant sign, the second moment is roughly $T\sigma^2+T^2m^2$ against $T\sigma^2$ with centering.

SFT is the extreme case. Its sampler is the data’s next-token distribution $q$, unknown since each prefix shows a single token, and its signal is constant, $A\equiv1$. The covariance is then zero and the average signal is one, so by the tower property the gradient is exactly the accumulated shift:

Centering would remove all of it, from either side: a constant signal is its own value baseline, and $\operatorname{Cov}_{\smu}(1,c_i)=0$. Knowing $q$ makes no difference, and centering with an estimate of $q$ keeps only the part of the shift that the estimate misses. Imitation is the mean shift, so how sparse SFT is depends on how far the data is from the model. The measurements agree: SFT on rejection-sampled, in-distribution data leaves 91% of parameters unchanged, SFT on out-of-distribution data 7% (Mukherjee et al., 2025). In the output layer a demonstrated token $y^\star$ that the model almost never produces gets a nearly full-size push, $(1-\pmodel(y^\star))\,h$, on a row whose on-policy Fisher is about $\pmodel(y^\star)$: data that leaves the model’s distribution touches parameters the model itself never touches.

4. Four cases side by side

The bound belongs to the estimator: it holds whenever centering removes the mean shift.

On-policy distillation (Agarwal et al., 2024) with a reverse-KL loss is policy gradient whose per-token reward is the teacher’s log-probability minus the model’s, so it shares the first row. With a forward-KL loss, the gradient at each prefix is the shift toward the teacher’s next-token distribution, and distillation behaves like SFT. Sharing a row means sharing the cap; how much of it an update uses depends on the signal. In a controlled comparison on Qwen3-8B, 77.2% of weights stay unchanged at bf16 precision under RL with verifiable rewards (RLVR; Lambert et al., 2024), 51.6% under on-policy distillation and 8.1% under SFT (Shen et al., 2026). RLVR and distillation both sample from the model; RLVR gives one coefficient per response, distillation one per token.

Offline data with a varying signal reaches the second row’s bound without knowing its sampler, once a per-prefix value baseline is subtracted from the signal.

TD weights a raw logit gradient by the TD error $\delta$, $\mathbb{E}[\delta\cdot\kappa_{i,a}]$, which keeps the common component, since for a Q-value a uniform shift carries the value level. The bound does not apply, which leaves open whether TD updates are sparse. DPO (direct preference optimization; Rafailov et al., 2023) on a fixed preference set also has an unknown sampler and behaves like SFT: sparse on in-distribution data, dense on out-of-distribution data (Mukherjee et al., 2025). What separates the first three rows is the mean shift: zero when the score is centered, the whole signal in SFT.

5. Training–inference mismatch

Under training–inference mismatch, centering is what keeps numerical noise out of the update.

When the sampler is an inference engine, $\smu\approx\pmodel$, and the shift $m_i$ comes only from numerical differences between two implementations. It carries no reward information, yet it accumulates along the sequence and pushes directions with near-zero on-policy Fisher. In the output layer the shift for row $c$ is $(\smu(c)-\pmodel(c))\,h$: where the engine draws a token more often than the model would, that row is pushed although its on-policy Fisher is near zero, consistent with the mismatch doing most of its damage on rare tokens (When Speed Kills Stability).

Centering removes the shift exactly, and on its own it matches or beats importance sampling under quantization (Marek & Ryabinin, 2026). Without the engine’s full distribution, per-token importance weights do the same job with only the drawn token’s two probabilities: with $\rho_t=\pmodel(y_t\mid s_t)/\smu(y_t\mid s_t)$, $\mathbb{E}_{\smu}\big[\rho_t(\kappa_{i,y_t}-\bar\kappa^{p}_i)\big]=\sum_a\pmodel(a)\big(\kappa_{i,a}-\bar\kappa^{p}_i\big)=0$ at every prefix. Clipping a token’s ratio breaks this; Trust Region Masking never clips a token and only drops whole sequences that left the trust region. A per-prefix value baseline removes the shift too, as far as it matches $V^{\mu}(s_t)$, which the per-token critic of PPO (Schulman et al., 2017) aims for; a baseline shared by the whole response, as in GRPO (Shao et al., 2024), leaves the shift weighted by $V^{\mu}(s_t)$ minus that baseline. Here sparsity means leaving alone the parameters that only noise would push.

6. Scope and open questions

• What centering needs. On the score side, the sampler’s full next-token distribution at every prefix (a stale model, or an engine that returns full log-probabilities; with top-k or top-p, the truncated distribution), or per-token importance weights, which need only the drawn token’s two probabilities. On the signal side, a per-prefix value baseline, which works on offline data with an unknown sampler as long as the signal varies. Demonstrations alone provide neither probabilities nor a varying signal. Estimating $\bar\kappa^{\mu}$ or $V^{\mu}$ from a batch makes the cancellation approximate.
• What it changes. Off-policy, centering drops the mean-shift term, which pulls the model toward the sampler; the coefficient $\beta$ in Around the Sampler, the average signal of section 3’s split, sets that pull on purpose.
• Two mechanisms, still entangled. A concentrated sampler and candidates that react alike both lower the Fisher, and the ratio in section 2 changes both at once. Cleaner tests: permute which tokens get which probabilities at fixed entropy, and measure, per coordinate, the share of the second moment that centering removes, $(\bar\kappa^{p}_i)^2/\sum_a\pmodel(a)\,\kappa_{i,a}^2$; a share near 1 on low-Fisher coordinates would mean the common part dominates.
• One step. The Fisher itself changes during training; Part 3 works out what a one-step bound implies.

Next. Part 3 turns the bound around: how far can one update move a response the model almost never produces?

References

• Y. Zhang, S. Mukherjee, H. Peng. Understanding LLM Parameter Update Sparsity through the Lens of Fisher. arXiv, 2026.
• S. Mukherjee, L. Yuan, D. Hakkani-Tür, H. Peng. Reinforcement Learning Finetunes Small Subnetworks in Large Language Models. NeurIPS 2025.
• Y. Li. Logit Dynamics in Softmax Policy Gradient Methods. arXiv, 2025.
• J. Liu, Y. Li, et al. When Speed Kills Stability: Demystifying RL Collapse from the Training-Inference Mismatch. Blog, 2025.
• Y. Li, J. Liu, et al. Trust Region Masking for Long-Horizon LLM Reinforcement Learning. ICML 2026.
• M. Marek, M. Ryabinin. Score Centering Stabilizes Off-policy Reinforcement Learning. arXiv, 2026.
• Z. Shen, Y. Li, Q. Yin, et al. On the Geometry of On-Policy Distillation. arXiv, 2026.
• Y. Wu, Y. Zhou, Z. Ziheng, et al. On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectification. arXiv, 2025.
• R. Agarwal, N. Vieillard, Y. Zhou, et al. On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes. ICLR 2024.
• N. Lambert, J. Morrison, V. Pyatkin, et al. Tülu 3: Pushing Frontiers in Open Language Model Post-Training. arXiv, 2024.
• R. Rafailov, A. Sharma, E. Mitchell, et al. Direct Preference Optimization: Your Language Model is Secretly a Reward Model. NeurIPS 2023.
• J. Schulman, F. Wolski, P. Dhariwal, et al. Proximal Policy Optimization Algorithms. arXiv, 2017.
• Z. Shao, P. Wang, Q. Zhu, et al. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv, 2024.

Citation

@online{li-2026-sparsity-score-centering,
  title  = {Sparsity Requires Score Centering},
  author = {Li, Yingru},
  year   = {2026},
  month  = oct,
  url    = {https://richardli.xyz/post/sparsity-score-centering/}
}