Regret comparable to exact Thompson sampling with only Θ(d log T) ensemble directions, kept by incremental updates; scales to online language tasks on a GPT-2 backbone.
Human-level Atari with 15% of DDQN's interaction data (1.5M), as a small addition to DQN.
A martingale extension of the Johnson–Lindenstrauss lemma to sequentially dependent, adaptively chosen data.
Traced LLM-RL collapse to the training–inference mismatch, an off-policy bias, and introduced masked importance sampling (MIS).
The principled form of MIS: a sequence-level trust-region mask with per-token importance sampling; a first-class algorithm in veRL.
The mismatch is not a static numerical error; it moves with the size of the update.
Safe reuse of off-policy data through a divergence between state distributions.
A per-token baseline weighted by realized gradient energy; group size 4 matches 32.
Importance sampling and masked importance sampling (MIS), at the token and sequence level, for general off-policy training.
A 14B kernel agent; ahead of Claude-4.5-Sonnet and GPT-5 on KernelBench Level 2 with test-time scaling.