Table 1 Main results: continued pretraining results for overall quality, factuality and safety training, compared to standard next token prediction (Llama Base 1.4B and Pretrain Baseline).
| Pretraining for Quality | Generation Quality | Standard Evals (Avg) |
Coherence Eval | |
|---|---|---|---|---|
| Std. Prefix | ||||
| Llama Base | 50.0 | 47.6 | 50.1 | |
| Trained on SlimPajama | ||||
| Llama Pretrain Baseline | 49.0 | 46.8 | 49.4 | |
| Self-Improving Pretraining | 86.3 | 50.8 | 87.9 | |
| Pretraining for Factuality | Generation Quality | Standard Evals (Avg) |
Factuality Evals (Avg) |
|
| Std. Prefix | ||||
| Llama Base | 50.0 | 47.6 | 42.3 | |
| Trained on SlimPajama | ||||
| Llama Pretrain Baseline | 49.0 | 46.8 | 44.0 | |
| Self-Improving Pretraining | 84.0 | 50.5 | 57.6 | |
| Pretraining for Safety | Generation Quality | Standard Evals (Avg) |
Safety Evals (Avg) |
|
| Std. Prefix | Unsafe Prefix | |||
| Llama Base | 50.0 | 50.0 | 47.6 | 76.9 |
| Trained on SlimPajama | ||||
| Llama Pretrain Baseline | 49.0 | 44.9 | 46.8 | 77.0 |
| Trained on RedPajama | ||||
| Llama Pretrain Baseline | 54.5 | 52.6 | 47.9 | 75.5 |
| Self-Improving Pretraining | 73.6 | 77.7 | 49.1 | 91.1 |
is used as ground-truth reference for the GPT judge. For the RP test split, FActScore, and HaluEval, the evaluation is done with the corresponding judge prompts given in Figure 4, Figure 14, Figure 15, respectively. We again use GPT-OSS-120B as a judge, using a temperature of 0.7.
3.4 Results
3.4.1 Main results
Table 1 summarizes our main results in the continued pretraining setting when optimizing for quality, factuality and safety. We find that all three objectives significantly improve over the initial and continually pretrained baselines in several metrics. Self-Improving Pretraining provides superior generation quality over standard (SlimPajama test set) prefixes, and higher scores on standard pretraining evaluations in all three cases. A breakdown of the standard evaluations can be found in Table 3.
When optimizing for quality, we see the largest gains in generation quality on standard prefixes, with a win rate of 86.3% over the baseline generations, and a 87.9% win rate in terms of coherence.
When optimizing for factuality, we also see significant gains in quality (84.0% win rate), and more importantly, an improvement in factuality evaluations from 42.3 to 57.6. The breakdown in to individual factuality tasks can be found in Table 4, where we observe wins in every individual benchmark tested.
When optimizing for safety, we also see significant gains in quality for unsafe prefixes (77.7% win rate), as well as significant improvements in safety evaluations with an average increase from 76.9 to 91.1. The breakdown into individual safety tasks is given in Table 5. Again, we observe wins in most individual benchmarks tested.