Policy Gradient

One policy-gradient update moves the log-odds between two responses by at most std(A) times their score difference measured in the sampler’s Fisher: a response moves only as far as it resembles what the sampler produces. With one logit per response, a rare response takes time proportional to one over its probability, while a demonstration needs only logarithmic time. The bound covers one step of one estimator; it does not say RL can only sharpen.

With a known sampler, sparsity requires the score to be centered under it; whether the data is on-policy matters only through that. A centered update is capped by the sampler’s Fisher, which is concentrated: in the output layer a row can move only if the sampler emits its token, and in hidden layers only the directions that tell likely candidates apart remain. Without centering, a mean shift adds up along the sequence and escapes the cap; under training–inference mismatch it is numerical noise. SFT cannot be centered, since its sampler is unknown: its gradient is the shift itself.

On the sampler’s own samples the score has zero mean, so the expected policy gradient is the covariance between the learning signal and the score, and the Fisher is the score’s variance. A parameter can move only where its Fisher is large; whether it does is up to the reward. Off-policy, all of this holds once the score is centered under the sampling distribution.

Given its history, a learner gains nothing from a response it sampled itself, so every bit enters through the reward. A pass/fail episode teaches at most one bit, and less when the outcome is predictable. With a reward at every token, one scalar advantage keeps at most log₂ B bits; per-token advantages keep the whole reward sequence.