Thompson sampling sends the learner where plausible models disagree, and stops exploring on its own once they agree. Its price is a fresh posterior sample every round. I build the randomness into the model instead: a random index as a second input, trained so that simple noise is pushed forward onto the posterior. A sample is one forward pass; an update is one SGD step that moves every sample at once. The theory rests on a sequential extension of the Johnson–Lindenstrauss lemma to data the agent chose itself.

• Scalable Exploration via Ensemble++
  NeurIPS 2025 arXiv Blog

  Regret comparable to exact Thompson sampling with only Θ(d log T) ensemble directions, kept by incremental updates.

• Q-Star Meets Scalable Posterior Sampling: Bridging Theory and Practice via HyperAgent
  ICML 2024 arXiv Slides

  Human-level Atari with 15% of DDQN's interaction data (1.5M), as a small addition to DQN.

• HyperDQN: A Randomized Exploration Method for Deep Reinforcement Learning
  ICLR 2022 OpenReview Slides
• Probability Tools for Sequential Random Projection
  arXiv 2024 arXiv

  A martingale extension of the Johnson–Lindenstrauss lemma to sequentially dependent, adaptively chosen data.

Large RL runs sample on fast inference engines and train elsewhere. The two engines assign different probabilities to the same tokens even at identical weights, and asynchrony adds stale data on top, so a run set up to be on-policy is off-policy anyway. We traced LLM-RL collapse to this training–inference mismatch, then built corrections that act on the whole sequence: sequence-level importance sampling and masking, a sequence-level trust region, and a learning-rate schedule that decays the step size as instability builds. Around the sampler, the gain of an update is a tangent plus a remainder bounded by the reward range times the sequence KL, and a batch-size-invariant rule for the step size and the queue keeps the gap fixed as runs grow.

• When Speed Kills Stability: Demystifying RL Collapse from the Training-Inference Mismatch
  Technical blog, 2025 Blog

  Traced LLM-RL collapse to the gap between the inference engine and the training engine.

• Trust Region Masking for Long-Horizon LLM Reinforcement Learning
  ICML 2026 arXiv Blog Slides

  A sequence-level trust region for long-horizon LLM-RL; a first-class algorithm in veRL.

• Beyond Precision: Training-Inference Mismatch is an Optimization Problem and Simple LR Scheduling Fixes It
  arXiv 2026 arXiv Blog

  The mismatch is not a static numerical error; it moves with the size of the update.

• Divergence-Augmented Policy Optimization
  NeurIPS 2019 arXiv

  Safe reuse of off-policy data through a divergence between state distributions.

Even fully on-policy, runs collapse, and a spike in gradient variance comes first: one sparse reward multiplies a sum of thousands of token gradients. The best baseline is a center of mass weighted by gradient energy. Energy builds up along the rollout, so the Optimal Token Baseline is per token, and it costs nothing extra: the logit-gradient norm is read off the forward pass.

• The Optimal Token Baseline: Variance Reduction for Long-Horizon LLM-RL
  ICML 2026 arXiv Blog veRL

  A per-token baseline weighted by realized gradient energy; group size 4 matches 32.

• Logit Dynamics in Softmax Policy Gradient Methods
  arXiv 2025 arXiv