PPO

Bias and variance of each off-policy estimator for LLM-RL: sequence-level IS is unbiased but its variance grows exponentially with length, token-level IS and PPO-style truncation trade that for an O(T²) bias, and truncated sequence-level IS balances the two.

Off-policy mismatch breaks RL in two distinct ways: bias, measured by total variation, pushes the optimizer toward the wrong solution, and variance, measured by chi-squared divergence, stalls progress. A stochastic-gradient-ascent framework for telling them apart.