| BoolQA | PIQA | Hellaswag | ARC-e | ARC-c | OBQA | SIQA | MMLU | |
|---|---|---|---|---|---|---|---|---|
| Llama Base | 64.6 | 74.8 | 47.9 | 66.6 | 32.3 | 27.2 | 41.0 | 26.4 |
| Trained on SlimPajama | ||||||||
| Llama Pretrain Baseline | 59.6 | 74.2 | 47.7 | 65.3 | 31.3 | 27.0 | 42.2 | 26.7 |
| Pretraining for Quality | ||||||||
| Self-Improving Pretraining | 69.1 | 75.8 | 51.7 | 69.4 | 35.7 | 30.0 | 46.1 | 28.3 |
| Pretraining for Factuality | ||||||||
| Self-Improving Pretraining | 70.3 | 75.1 | 51.1 | 69.1 | 35.1 | 29.0 | 46.8 | 27.9 |
| Pretraining for Safety | ||||||||
| Trained on RedPajama | ||||||||
| Llama Pretrain Baseline | 64.0 | 74.3 | 49.2 | 66.9 | 32.8 | 26.6 | 41.5 | 27.5 |
| Self-Improving Pretraining | 65.7 | 75.6 | 49.6 | 69.0 | 34.8 | 27.4 | 44.1 | 26.7 |
| Pretraining for Factuality | SlimPajama (pointwise) |
FActScore (pairwise) |
HaluEval dialogue |
HaluEval QA |
HaluEval summarization |
Truthful QA MC1 |
TruthfulQA MC2 |
|---|---|---|---|---|---|---|---|
| Llama Base | 36.6 | 50.0 | 50.0 | 50.1 | 50.0 | 22.4 | 35.9 |
| Trained on SlimPajama | |||||||
| Llama Pretrain Baseline | 35.4 | 48.9 | 50.8 | 51.4 | 61.5 | 21.5 | 35.5 |
| Self-Improving Pretraining | 63.5 | 69.3 | 54.6 | 58.5 | 84.7 | 27.7 | 42.5 |
to 75.5), while standard evaluations are similar or slightly improved (47.6 vs. 47.9). As RedPajama contains unsafe contexts this is not unexpected. Continued pretraining on the cleaner SlimPajama keeps the safety evaluations more or less unchanged (76.9 vs. 77.0), although standard evaluations drop.
Next, training with SFT on rewrites or a single rollout without a judge gives little improvement in quality for the former (52.7 of safe and 50.6 on unsafe prefixes), and large deterioration for the latter (dropping to 2.0 and 0.2 on safe and unsafe prefixes), which is expected (i.e., model collapse). Upon inspection of the model generations, we found that the model trained on a single rollout collapsed to generating meaningless - but safe - sequences of words or symbols. In contrast online DPO with the rewrite as chosen and current rollout as rejected gives slightly improved standard and safety evaluations (48.8 and 77.7 respectively).
Overall, however, with our full Self-Improving Pretraining method using a post-trained suffix judge, we find much larger gains – particularly in the online DPO case, and for larger numbers of rollouts. We find applying RF-NLL improves safety evaluations over the baseline (85.0 vs. 76.9) but is only on par with the improvement found using SFT on rewrites, which does not use a judge, while both do not give significant gains in generation quality. For online DPO however, we see major boosts in generation quality. Online DPO using rewrites and a single rollout improves generation quality from 50.0 to 60.0 on standard prefixes, and from 50.0 to 87.2 on unsafe prefixes. Increasing to 16 rollouts gives even larger gains on standard prefixes (from 50.0 to 73.6), and on overall safety evaluations (from 76.9 to 91.1).
3.5.2 Suffix & rewrite vs. rollouts
In both the continual and from-scratch pretraining settings, we find that early in training the model relies on the original and rewritten suffixes more often for supervision. As the model improves the judge picks rollouts more and more frequently, see Figure 8. Later in training RL rewards high-quality rollouts, resulting in a higher rollout chosen rate.