Training Dynamics

2026

  • Keeping the Off-Policy Gap Fixed as RL Scales Off-Policy RL at Scale · Part 2 of 2

    Asynchronous RL buys throughput with fast kernels, a queue and larger batches, and pays for it in the gap between sampler and trainer. Model the gap as an engine-mismatch floor plus staleness, put everything on a data clock, and two settings keep it fixed as the batch grows: the step size follows the optimizer, and the queue is sized in rollouts.

  • Around the Sampler: Tangent, Remainder and KL Off-Policy RL at Scale · Part 1 of 2

    Measured from the sampler, the gain of any policy update is a tangent plus a remainder. The tangent is a covariance loss whose gradient needs only the trainer; the remainder is bounded by R_max times the sequence KL, with no horizon factor. Score centering, per-token importance sampling, KL penalties and sequence masking all fall out of this one picture.

2025

  • The Stability Gap: Why Top-K Routing Breaks RL Optimization

    A rigorous mathematical analysis showing that Top-K expert routing in Mixture of Experts creates two fundamental pathologies: gradient blackout (zero gradients almost everywhere) and first-order approximation failure (discontinuous policy mapping), explaining why MoE-RL training can be unstable.

  • When Speed Kills Stability: Demystifying RL Collapse from the Training-Inference Mismatch

    The relentless push for faster inference creates a dangerous training-inference mismatch that silently kills RL with LLMs. We trace how it leads to collapse, most acutely in reasoning and agentic RL, and propose masked importance sampling (MIS): one importance ratio per response, with sequences outside the trust region masked out of the loss. Its length-normalized form, Geo-MIS, masks on the geometric mean of the per-token ratios, a one-sample estimate of the mean per-token KL that Trust Region Masking later computes exactly. In our multi-turn tool-use runs, token-level corrections still collapsed, while MIS trained past 700 steps where the uncorrected run collapsed near step 280.

All tags