Exploration

2026

  • What Policy Gradient Can Reach The Shape of a Post-Training Update · Part 3 of 4

    Per unit step of policy gradient, the log-odds between two responses change by at most std(A) times their score difference measured in the sampler's Fisher: a response moves only as far as it resembles what the sampler produces. With one logit per response, a rare response takes time proportional to one over its probability, while a demonstration needs only logarithmic time. The bound covers one step of one estimator; shared structure, many steps and a different sampler can all widen what policy gradient reaches.

  • Why d log T Particles Are Enough, and What Carries Over to Networks Exploration at Scale · Part 2 of 2

    The particles of Ensemble++ are a random projection of the data history, so tracking the posterior is a Johnson–Lindenstrauss statement. When the agent picks its own data the classical lemma fails; a sequential version survives at the price of one log T, so d log T particles suffice. In a network the same update is an SGD step on M linear heads.

  • Thompson Sampling for the Price of an SGD Step Exploration at Scale · Part 1 of 2

    Thompson sampling explores where plausible models disagree, but needs a fresh posterior sample every round. Build the randomness into the model instead: a random index as a second input, and a cloud of particles that moves with every data point. A sample becomes one forward pass, and the update is ordinary incremental learning.

All tags