Score Centering

Sparsity needs two ingredients. A score centered under the sampler makes the Fisher a cap on each parameter’s update, and the Fisher is concentrated when the sampler puts its mass on a few likely tokens that react alike. Without centering, a mean shift escapes the cap. Supervised fine-tuning is all mean shift, because its signal is constant, so centering would leave nothing to learn.

On the model’s own samples the score has zero mean, so the expected policy gradient is the covariance between the learning signal and the score, and the Fisher is the score’s variance. A parameter can move only if its Fisher is large; whether it does is up to the reward. Off-policy, all of this holds once the score is centered under the sampler.