Exploration

Per unit step of policy gradient, the log-odds between two responses change by at most std(A) times their score difference measured in the sampler’s Fisher: a response moves only as far as it resembles what the sampler produces. With one logit per response, a rare response takes time proportional to one over its probability, while a demonstration needs only logarithmic time. The bound covers one step of one estimator; shared structure, many steps and a different sampler can all widen what policy gradient reaches.

The particles of Ensemble++ are a random projection of the data history, so tracking the posterior is a Johnson–Lindenstrauss statement. When the agent picks its own data the classical lemma fails; a sequential version survives at the price of one log T, so d log T particles suffice. In a network the same update is an SGD step on M linear heads.

Thompson sampling explores where plausible models disagree, but needs a fresh posterior sample every round. Build the randomness into the model instead: a random index as a second input, and a cloud of particles that moves with every data point. A sample becomes one forward pass, and the update is ordinary incremental learning.