Reinforcement Learning

At the output layer only the rows of tokens that compete can move. The signal that backpropagation carries into the network is the drawn token’s output vector minus the sampler’s average, so it lies in the span of differences of those vectors and vanishes where the sampler is sure. A linear map that feeds the unembedding therefore gets an expected update of rank at most the number of competing tokens minus one, plus a leak set by the tail mass, and a LoRA of that rank carries the main part. Measured updates are numerically full rank yet spectrally concentrated; the optimizer and many steps account for the difference.

Per unit step of policy gradient, the log-odds between two responses change by at most std(A) times their score difference measured in the sampler’s Fisher: a response moves only as far as it resembles what the sampler produces. With one logit per response, a rare response takes time proportional to one over its probability, while a demonstration needs only logarithmic time. The bound covers one step of one estimator; shared structure, many steps and a different sampler can all widen what policy gradient reaches.

Sparsity needs two ingredients. A score centered under the sampler makes the Fisher a cap on each parameter’s update, and the Fisher is concentrated when the sampler puts its mass on a few likely tokens that react alike. Without centering, a mean shift escapes the cap. Supervised fine-tuning is all mean shift, because its signal is constant, so centering would leave nothing to learn.

On the model’s own samples the score has zero mean, so the expected policy gradient is the covariance between the learning signal and the score, and the Fisher is the score’s variance. A parameter can move only if its Fisher is large; whether it does is up to the reward. Off-policy, all of this holds once the score is centered under the sampler.

The particles of Ensemble++ are a random projection of the data history, so tracking the posterior is a Johnson–Lindenstrauss statement. When the agent picks its own data the classical lemma fails; a sequential version survives at the price of one log T, so d log T particles suffice. In a network the same update is an SGD step on M linear heads.

Thompson sampling explores where plausible models disagree, but needs a fresh posterior sample every round. Build the randomness into the model instead: a random index as a second input, and a cloud of particles that moves with every data point. A sample becomes one forward pass, and the update is ordinary incremental learning.

Asynchronous RL buys throughput with fast kernels, a queue and larger batches, and pays for it in the gap between sampler and trainer. Model the gap as an engine-mismatch floor plus staleness, put everything on a data clock, and two settings keep it fixed as the batch grows: the step size follows the optimizer, and the queue is sized in rollouts.

Measured from the sampler, the gain of any policy update is a tangent plus a remainder. The tangent is a covariance loss whose gradient needs only the trainer; the remainder is bounded by R_max times the sequence KL, with no horizon factor. Score centering, per-token importance sampling, KL penalties and sequence masking all fall out of this one picture.

RL training for LLMs is notoriously unstable. While recent studies attribute this to training-inference mismatch from hybrid engines, we show this is not merely a static numerical issue, but a dynamic problem coupled with the model’s optimization trajectory. We propose a specialized Learning Rate Scheduler that decays LR as gradient noise rises, using response length surge as a reliable early indicator of impending instability.

RL training for LLMs frequently suffers from training collapse due to exploding gradient variance in long-horizon tasks. We derive the Optimal Token Baseline (OTB) from first principles, proving that updates should be weighted inversely to their accumulated uncertainty (Realized Energy). Our computationally free Logit-Gradient Proxy eliminates training collapse, matches N=32 performance with just N=4, and reduces token consumption by 62-66%.