Masked Importance Sampling

We derive tighter off-policy bounds for LLM-RL: O(T^1.5) Pinsker-Marginal and O(T) Mixed bounds, compared to classical O(T²). We propose Trust Region Masking (TRM), which excludes entire sequences from gradient computation if any token violates the trust region. TRM is the principled form of masked importance sampling (MIS): a sequence-level trust-region mask combined with per-token importance sampling.

Hard trust regions for long-horizon LLM-RL: sequence-level masked importance sampling (Seq-MIS) rejects out-of-distribution samples instead of clipping them, and geometric masking makes the trust region length-invariant.

The relentless push for faster inference creates a dangerous training-inference mismatch that silently kills RL with LLMs. We reveal the vicious cycle—particularly acute in reasoning and agentic RL—and introduce masked importance sampling (MIS), sequence-level importance sampling that masks rollouts outside the trust region.