Trust Region

Measured from the sampler, the gain of any policy update is a tangent plus a remainder. The tangent is a covariance loss whose gradient needs only the trainer; the remainder is bounded by R_max times the sequence KL, with no horizon factor. Score centering, per-token importance sampling, KL penalties and sequence masking all fall out of this one picture.

We derive tighter off-policy bounds for LLM-RL: O(T^1.5) Pinsker-Marginal and O(T) Mixed bounds, compared to classical O(T²). We propose Trust Region Masking (TRM), which excludes entire sequences from gradient computation if any token violates the trust region. TRM is the principled form of masked importance sampling (MIS): a sequence-level trust-region mask combined with per-token importance sampling.

Hard trust regions for long-horizon LLM-RL: sequence-level masked importance sampling (Seq-MIS) rejects out-of-distribution samples instead of clipping them, and geometric masking makes the trust region length-invariant.