Importance Sampling

Definitive mathematical formulations for rollout correction methods in VeRL, progressing from REINFORCE to PPO to Decoupled PPO. Handles policy mismatch, temporal lag, replay buffers, and off-policy algorithms with importance sampling and rejection sampling techniques.

Bias and variance of each off-policy estimator for LLM-RL: sequence-level IS is unbiased but its variance grows exponentially with length, token-level IS and PPO-style truncation trade that for an O(T²) bias, and truncated sequence-level IS balances the two.

The relentless push for faster inference creates a dangerous training-inference mismatch that silently kills RL with LLMs. We reveal the vicious cycle—particularly acute in reasoning and agentic RL—and introduce masked importance sampling (MIS), sequence-level importance sampling that masks rollouts outside the trust region.