Figure 8 Rollout chosen rate on the training data during from-scratch pretraining (left) and continued pretraining (right). Initially RL reward for rollouts is low, and suffix or rewrite completions are chosen for training more often. As the model improves, RL rewards high-quality rollouts, resulting in higher rollout chosen rates.
Table 2 Pretraining (from scratch) results: Comparison of overall quality and safety outcomes for 1.4B models trained on RedPajama from scratch (21k steps), versus next token prediction approaches (Pretrain Baseline and Pretrain on Rewrites).
| Pretraining for Safety (from scratch) | Generation Quality | Safety Evals (Avg) |
|
|---|---|---|---|
| Std. Prefix | Unsafe Prefix | ||
| Pretrain Baseline | 1.3 | 2.4 | 85.2 |
| Pretrain on Rewrites | 1.6 | 2.4 | 96.7 |
| Self-Improving Pretraining: RF-NLL (suffix vs. rewrite) | 5.3 | 25.8 | 96.4 |
| Self-Improving Pretraining: RF-NLL (rollout vs. rewrite) | 32.4 | 12.1 | 97.5 |
3.4.2 Pretraining from-scratch results
The previous results are from continued pretraining from the initial Llama baseline model. Potentially, our Self-Improving Pretraining could provide much larger improvements if used earlier in pretraining, for example by making the model learn safety measures earlier on in training.
We compare 4 training setups in the safety pretraining setting:
- • Pretrain Baseline (model trained on RedPajama suffixes);
- • Pretrain on Rewrites;
- • Self-Improving Pretraining: RF-NLL (suffix vs. rewrite);
- • Self-Improving Pretraining: RF-NLL (rollout vs. rewrite).
In these experiments, we only use 1 rollout for training.
Table 2 summarizes quality and safety evaluation results. NLL pretraining on rewritten suffixes outperforms baseline training on safety evaluations, but does not improve on overall quality. Using the fine-tuned Llama3.1- 8B-Instruct suffix judge promotes generations that are better in both quality and in safety, resulting in improved performance for our models. Self-Improving Pretraining using RF-NLL (rollout vs. rewrite) has a generation quality win rate of 32.4, compared to the next-token prediction baseline win rate of only 1.3 – a huge improvement. Simultaneously, safety evaluations improve from 85.2 to 97.5.
3.5 Analysis & ablations
3.5.1 Training objective
Table 6 provides ablation results on variants of the Self-Improving Pretraining training objective in the safety optimization case. First, we find that continued pretraining using standard next token prediction on RedPajama lowers the performance compared to the initial baseline on safety evaluations slightly (from 76.9