Trust Region

2026

  • Around the Sampler: Tangent, Remainder and KL Off-Policy RL at Scale · Part 1 of 2

    Measured from the sampler, the gain of any policy update is a tangent plus a remainder. The tangent is a covariance loss whose gradient needs only the trainer; the remainder is bounded by R_max times the sequence KL, with no horizon factor. Score centering, per-token importance sampling, KL penalties and sequence masking all fall out of this one picture.

2025

  • Trust Region Masking for Long-Horizon LLM Reinforcement Learning

    We derive tighter off-policy bounds for LLM-RL: O(T^1.5) Pinsker-Marginal and O(T) Mixed bounds, compared to classical O(T²). We propose Trust Region Masking (TRM), which excludes entire sequences from gradient computation if any token violates the trust region. TRM is the principled form of masked importance sampling (MIS): a sequence-level trust-region mask combined with per-token importance sampling.

  • Trust Region Optimization via Sequence Masking RL Collapse · Part 3 of 3

    Hard trust regions for long-horizon LLM-RL: sequence-level masked importance sampling (Seq-MIS) rejects out-of-distribution samples instead of clipping them, and geometric masking makes the trust region length-invariant.

All tags