Learning to summarize from human feedback

SFT and Fidelity

The issue with SFT is that it is not good at learning fidelity. It teaches the model to copy the exact tokens of an answer. For example, suppose X is one valid word and Y is equally probable. SFT may penalize Y even when Y is equally correct. RLHF teaches the model that multiple outputs can accomplish the same thing while still learning the right behavior. The model needs to learn the underlying semantics, not the exact word.

SFT Treats All Demonstrations as Correct

Models are incentivized to place probability mass on all human demonstrations, including those that are low-quality.

Suppose your dataset contains three summaries for similar posts:

• Summary A: Excellent
• Summary B: Good
• Summary C: Mediocre

SFT doesn't know that A is better than C.

It simply learns:

• Increase $P(A)$
• Increase $P(B)$
• Increase $P(C)$

because all demonstrations are treated as correct.

But what we actually want is:

$P(A)>P(B)>P(C)$

SFT has no way of expressing this preference.

RLHF fixes this because humans can say:

• $A>B$
• $A>C$
• $B>C$

The model then learns to prefer better outputs.

Distributional Shift During Sampling

Distributional shift during sampling can degrade performance.

During training, the human-written sequence is:

Human: Token1 → Token2 → Token3 → Token4

The model always conditions on the correct previous tokens.

Beam Search

I was not aware of beam search before this paper introduced it to me. At inference time, instead of picking a single token, you pick several candidate tokens, for example, two. You branch from each candidate and continue generating, which gives you many sequences.

You calculate the cumulative probability for each sequence and choose the one with the highest total score. We do this because we do not want to maximize only the probability of the earliest token; we want to maximize the probability of the entire sequence.

The issue is that the sequence with the highest probability is not necessarily the correct answer. The model might simply repeat tokens. For example, if 'teacher' has the highest probability, it might produce 'teacher teacher teacher teacher teacher...'. That is why beam search was new to me: this paper introduced it to me.

Generalization

SFT did not generalize, but RL did. They trained on Reddit summaries, and summaries of news articles also improved with RL.

Reward Hacking from Over-Optimization

The authors test what happens when they optimize harder and harder.

Initially, a higher reward-model score means better summaries according to humans:

Higher reward-model score $\Rightarrow$ better summaries according to humans.

But after too much optimization:

Higher reward-model score $\nRightarrow$ better summaries.

Eventually, the policy discovers weaknesses in the reward model and gets a high predicted reward while producing worse text. This is reward hacking, or reward-model overoptimization.

Iterative Reward Modeling and PPO

The most surprising thing in this paper is how they train the reward models over time. You might think they take the Reddit summaries, generate summaries with the pretrained model and the supervised fine-tuned model, and train a reward model from those outputs using DPO or something similar. But that's not how they do it.

First, they train a reward model. Then they run PPO. Next, they train another reward model that also includes the PPO-generated answers. They keep repeating this process. The reward models keep improving because they receive improved PPO summaries. They also send the summaries to humans for a baseline.

They do one more cool thing: top-N sampling. In the supervised fine-tuned model, they generate N summaries. Then the reward model scores those N summaries, and they select the top-ranked summaries.

Testing Reward-Model Overoptimization

They wanted to test how long PPO could be run before it reward-hacked the reward model. The PPO reward was the reward-model score minus $\beta$ times the KL divergence from the SFT policy:

$R_{\text{PPO}}=R_{\text{RM}}-\beta D_{\text{KL}}(\pi \Vert {\pi }_{\text{SFT}})$

They vary $\beta$ and observe how the policy changes and when reward hacking appears. They compare the results with a human baseline.

Human Evaluation

To evaluate the model, they show human raters two summaries: the original Reddit summary and the summary generated by the PPO model. They ask which summary is better. The final score is the percentage of comparisons in which the PPO-generated summary is chosen over the human summary.

Training the Reward Model

For a pair of summaries $y_i$ and $y_j$, where $y_i$ is the better summary, they pass both through the reward model and obtain rewards $R_i$ and $R_j$. They calculate the difference between the rewards and apply the sigmoid function.

If the difference is positive, the resulting probability is greater than 0.5. If it is negative, the probability is less than 0.5. Because we know that $y_i$ is the preferred summary, we maximize the probability of this ordering.

The loss becomes:

$\mathcal{L}=-\log \sigma (R_i-R_j)$

They also calculate the average reward assigned to human summaries after the reward model is trained. They use this average as a baseline and want it to be zero. In the future, they subtract this average from every reward:

$R^{\prime }(y)=R(y)-{\bar{R}}_{\text{human}}$