Language Models

One policy-gradient update moves the log-odds between two responses by at most std(A) times their score difference measured in the sampler’s Fisher: a response moves only as far as it resembles what the sampler produces. With one logit per response, a rare response takes time proportional to one over its probability, while a demonstration needs only logarithmic time. The bound covers one step of one estimator; it does not say RL can only sharpen.

With a known sampler, sparsity requires the score to be centered under it; whether the data is on-policy matters only through that. A centered update is capped by the sampler’s Fisher, which is concentrated: in the output layer a row can move only if the sampler emits its token, and in hidden layers only the directions that tell likely candidates apart remain. Without centering, a mean shift adds up along the sequence and escapes the cap; under training–inference mismatch it is numerical noise. SFT cannot be centered, since its sampler is unknown: its gradient is the shift itself.

On the sampler’s own samples the score has zero mean, so the expected policy gradient is the covariance between the learning signal and the score, and the Fisher is the score’s variance. A parameter can move only where its Fisher is large; whether it does is up to the reward. Off-policy, all of this holds once the score is centered under the sampling distribution.

Measured from the sampler, the gain of any policy update is a tangent plus a remainder. The tangent is a covariance loss whose gradient needs only the trainer; the remainder is bounded by R_max times the sequence KL, with no horizon factor. Score centering, per-token importance sampling, KL penalties and sequence masking all fall out of this one picture.

RL training for LLMs is notoriously unstable. While recent studies attribute this to training-inference mismatch from hybrid engines, we show this is not merely a static numerical issue, but a dynamic problem coupled with the model’s optimization trajectory. We propose a specialized Learning Rate Scheduler that decays LR as gradient noise rises, using response length surge as a reliable early indicator of impending instability.

RL training for LLMs frequently suffers from training collapse due to exploding gradient variance in long-horizon tasks. We derive the Optimal Token Baseline (OTB) from first principles, proving that updates should be weighted inversely to their accumulated uncertainty (Realized Energy). Our computationally free Logit-Gradient Proxy eliminates training collapse, matches N=32 performance with just N=4, and reduces token consumption by 62-66%.

We derive tighter off-policy bounds for LLM-RL: O(T^1.5) Pinsker-Marginal and O(T) Mixed bounds, compared to classical O(T²). We propose Trust Region Masking (TRM), which excludes entire sequences from gradient computation if any token violates the trust region. TRM is the principled form of masked importance sampling (MIS): a sequence-level trust-region mask combined with per-token importance sampling.

Definitive mathematical formulations for rollout correction methods in VeRL, progressing from REINFORCE to PPO to Decoupled PPO. Handles policy mismatch, temporal lag, replay buffers, and off-policy algorithms with importance sampling and rejection sampling techniques.

Given its history, a learner gains nothing from a response it sampled itself, so every bit enters through the reward. A pass/fail episode teaches at most one bit, and less when the outcome is predictable. With a reward at every token, one scalar advantage keeps at most log₂ B bits; per-token advantages keep the whole reward sequence.

The relentless push for faster inference creates a dangerous training-inference mismatch that silently kills RL with LLMs. We reveal the vicious cycle—particularly acute in reasoning and agentic RL—and introduce masked importance sampling (MIS), sequence-level importance sampling that masks rollouts outside the trust region.