Score Centering

With a known sampler, sparsity requires the score to be centered under it; whether the data is on-policy matters only through that. A centered update is capped by the sampler’s Fisher, which is concentrated: in the output layer a row can move only if the sampler emits its token, and in hidden layers only the directions that tell likely candidates apart remain. Without centering, a mean shift adds up along the sequence and escapes the cap; under training–inference mismatch it is numerical noise. SFT cannot be centered, since its sampler is unknown: its gradient is the shift itself.

On the sampler’s own samples the score has zero mean, so the expected policy gradient is the covariance between the learning signal and the score, and the Fisher is the score’s variance. A parameter can move only where its Fisher is large; whether it does is up to the reward. Off-policy, all of this holds once the score is centered under the sampling distribution.