At the output layer only the rows of tokens that compete can move. The signal that backpropagation carries into the network is the drawn token's output vector minus the sampler's average, so it lies in the span of differences of those vectors and vanishes where the sampler is sure. A linear map that feeds the unembedding therefore gets an expected update of rank at most the number of competing tokens minus one, plus a leak set by the tail mass, and a LoRA of that rank carries the main part. Measured updates are numerically full rank yet spectrally concentrated; the optimizer and many steps account for the difference.
Per unit step of policy gradient, the log-odds between two responses change by at most std(A) times their score difference measured in the sampler's Fisher: a response moves only as far as it resembles what the sampler produces. With one logit per response, a rare response takes time proportional to one over its probability, while a demonstration needs only logarithmic time. The bound covers one step of one estimator; shared structure, many steps and a different sampler can all widen what policy gradient reaches.
Sparsity needs two ingredients. A score centered under the sampler makes the Fisher a cap on each parameter's update, and the Fisher is concentrated when the sampler puts its mass on a few likely tokens that react alike. Without centering, a mean shift escapes the cap. Supervised fine-tuning is all mean shift, because its signal is constant, so centering would leave nothing to learn.
On the model's own samples the score has zero mean, so the expected policy gradient is the covariance between the learning signal and the score, and the Fisher is the score's variance. A parameter can move only if its Fisher is large; whether it does is up to the reward. Off-policy, all of this holds once the score is centered under the sampler.
The particles of Ensemble++ are a random projection of the data history, so tracking the posterior is a Johnson–Lindenstrauss statement. When the agent picks its own data the classical lemma fails; a sequential version survives at the price of one log T, so d log T particles suffice. In a network the same update is an SGD step on M linear heads.
Thompson sampling explores where plausible models disagree, but needs a fresh posterior sample every round. Build the randomness into the model instead: a random index as a second input, and a cloud of particles that moves with every data point. A sample becomes one forward pass, and the update is ordinary incremental learning.
Asynchronous RL buys throughput with fast kernels, a queue and larger batches, and pays for it in the gap between sampler and trainer. Model the gap as an engine-mismatch floor plus staleness, put everything on a data clock, and two settings keep it fixed as the batch grows: the step size follows the optimizer, and the queue is sized in rollouts.
Measured from the sampler, the gain of any policy update is a tangent plus a remainder. The tangent is a covariance loss whose gradient needs only the trainer; the remainder is bounded by R_max times the sequence KL, with no horizon factor. Score centering, per-token importance sampling, KL penalties and sequence masking all fall out of this one picture.
RL training for LLMs is notoriously unstable. While recent studies attribute this to training-inference mismatch from hybrid engines, we show this is not merely a static numerical issue, but a dynamic problem coupled with the model's optimization trajectory. We propose a specialized Learning Rate Scheduler that decays LR as gradient noise rises, using response length surge as a reliable early indicator of impending instability.
RL training for LLMs frequently suffers from training collapse due to exploding gradient variance in long-horizon tasks. We derive the Optimal Token Baseline (OTB) from first principles, proving that updates should be weighted inversely to their accumulated uncertainty (Realized Energy). Our computationally free Logit-Gradient Proxy eliminates training collapse, matches N=32 performance with just N=4, and reduces token consumption by 62-66%.
We derive tighter off-policy bounds for LLM-RL: O(T^1.5) Pinsker-Marginal and O(T) Mixed bounds, compared to classical O(T²). We propose Trust Region Masking (TRM), which excludes entire sequences from gradient computation if any token violates the trust region. TRM is the principled form of masked importance sampling (MIS): a sequence-level trust-region mask combined with per-token importance sampling.
A rigorous mathematical analysis showing that Top-K expert routing in Mixture of Experts creates two fundamental pathologies: gradient blackout (zero gradients almost everywhere) and first-order approximation failure (discontinuous policy mapping), explaining why MoE-RL training can be unstable.
This post establishes a formal mathematical framework for language agents, deriving fundamental challenges from first principles and providing concrete design guidelines with real-world examples from SWE-Bench.
Definitive mathematical formulations for rollout correction methods in VeRL, progressing from REINFORCE to PPO to Decoupled PPO. Handles policy mismatch, temporal lag, replay buffers, and off-policy algorithms with importance sampling and rejection sampling techniques.
Hard trust regions for long-horizon LLM-RL: sequence-level masked importance sampling (Seq-MIS) rejects out-of-distribution samples instead of clipping them, and geometric masking makes the trust region length-invariant.
Bias and variance of each off-policy estimator for LLM-RL: sequence-level IS is unbiased but its variance grows exponentially with length, token-level IS and PPO-style truncation trade that for an O(T²) bias, and truncated sequence-level IS balances the two.
Off-policy mismatch breaks RL in two distinct ways: bias, measured by total variation, pushes the optimizer toward the wrong solution, and variance, measured by chi-squared divergence, stalls progress. A stochastic-gradient-ascent framework for telling them apart.
Given its history, a learner gains nothing from a response it sampled itself, so every bit enters through the reward. A pass/fail episode teaches at most one bit, and less when the outcome is predictable. With a reward at every token, one scalar advantage keeps at most log₂ B bits; per-token advantages keep the whole reward sequence.
The relentless push for faster inference creates a dangerous training-inference mismatch that silently kills RL with LLMs. We trace how it leads to collapse, most acutely in reasoning and agentic RL, and propose masked importance sampling (MIS): one importance ratio per response, with sequences outside the trust region masked out of the loss. Its length-normalized form, Geo-MIS, masks on the geometric mean of the per-token ratios, a one-sample estimate of the mean per-token KL that Trust Region Masking later computes exactly. In our multi-turn tool-use runs, token-level corrections still collapsed, while MIS trained past 700 steps where the uncorrected run collapsed near step 280.