Asynchronous Training
We tackle the fundamental challenge in this regime: how much off-policyness can we tolerate in asynchronous training so that we can speed up learning while maintaining performance?
The first thing that happened while reading this paper was that I read this line and immediately thought, "Oh, that means inference is generally slower than training." But that's not true.
While the trainer is learning, the generators do not stop. They keep producing data.
The catch is that, after a while:
- The generator is still using policy v10.
- The trainer has already updated to v11.
This is the issue.
REINFORCE Leave-One-Out
The first thing that surprised me while reading this paper was that there is an algorithm called REINFORCE Leave-One-Out. It is the same idea as GRPO. Exactly the same idea. The only difference is that it does not normalize by taking the mean and standard deviation. Other than that, it averages the samples and derives the baselines. I don't know why people don't take it seriously.
DPO
DPO has two forms.
Offline DPO
Offline DPO does not need a reward model. We start with preference pairs, and . We can calculate the probability of each answer given : and . We calculate these probabilities by multiplying the probabilities of the individual tokens.
Then we take the logarithm of both probabilities, subtract them, and apply the sigmoid function. At a high level, this gives us the preference objective that we maximize for the policy. This is the same preference information used to train the reward model.
Online DPO
With online DPO, we train in a similar way, but we are not limited to the dataset we already have. We also train a reward model and collect more data. The policy generates two answers, and the reward model judges them. Because we have the probabilities of both answers, we can optimize the policy to increase the preference gap between them.