Table 14 Ablation on rollouts: overall metrics. Evaluation results for quality, factuality and safety experiments with different numbers of rollouts.
| Pretraining for Quality | Generation Quality | Standard Evals (Avg) | Coherence Eval |
|---|---|---|---|
| Llama Base | 50.0 | 47.6 | 50.0 |
| Llama Pretrain Baseline | 49.0 | 46.7 | 49.4 |
| Self-Improving Pretraining (2 rollouts) | 69.9 | 49.4 | 67.6 |
| Self-Improving Pretraining (4 rollouts) | 75.5 | 49.9 | 71.2 |
| Self-Improving Pretraining (8 rollouts) | 84.3 | 51.1 | 86.8 |
| Self-Improving Pretraining (16 rollouts) | 86.3 | 50.8 | 87.9 |
| Pretraining for Factuality | Generation Quality | Standard Evals (Avg) | Factuality Evals (Avg) |
| Llama Base | 50.0 | 47.6 | 42.3 |
| Llama Pretrain Baseline | 49.6 | 46.8 | 44.0 |
| Self-Improving Pretraining (2 rollouts) | 65.2 | 49.0 | 46.2 |
| Self-Improving Pretraining (4 rollouts) | 69.0 | 49.7 | 48.8 |
| Self-Improving Pretraining (8 rollouts) | 83.1 | 50.3 | 56.9 |
| Self-Improving Pretraining (16 rollouts) | 84.0 | 50.5 | 57.6 |
| Pretraining for Safety | Gen. Quality (SP/RP) | Standard Evals (Avg) | Safety Evals (Avg) |
| Llama Base | 50.0 / 50.0 | 47.6 | 76.9 |
| Self-Improving Pretraining (rollout vs suf) | 55.7 / 84.7 | 48.4 | 82.5 |
| Self-Improving Pretraining (2 rollouts vs suf) | 57.3 / 85.0 | 47.4 | 86.2 |
| Self-Improving Pretraining (4 rollouts vs suf) | 63.0 / 69.9 | 48.3 | 85.2 |
| Self-Improving Pretraining (8 rollouts vs suf) | 69.0 / 73.8 | 48.3 | 85.4 |
| Self-Improving Pretraining (16 rollouts vs suf) | 73.6 / 77.7 | 49.1 | 91.1 |
| Self-Improving Pretraining (rollout vs suf vs rewr) | 58.1 / 52.3 | 48.6 | 88.9 |
| Self-Improving Pretraining (2 rollouts vs suf vs rewr) | 57.8 / 89.4 | 48.6 | 86.6 |
| Self-Improving Pretraining (4 rollouts vs suf vs rewr) | 62.3 / 68.6 | 48.9 | 88.4 |
| Self-Improving Pretraining (8 rollouts vs suf vs rewr) | 66.5 / 72.3 | 48.7 | 89.7 |
| Self-Improving Pretraining (16 rollouts vs suf vs rewr) | 72.5 / 75.4 | 49.1 | 88.9 |
Table 15 Ablation on rollouts: standard task metrics. Standard task results for quality and factuality training with different number of rollouts.
| boolq | piqa | siqa | hellaswag | arc_challenge | arc_easy | obqa | mmlu | |
|---|---|---|---|---|---|---|---|---|
| Llama Base | 64.6 | 74.8 | 41.0 | 47.9 | 32.3 | 66.6 | 27.2 | 26.4 |
| Pretraining for Quality | ||||||||
| Pretrain Baseline | 59.8 | 74.2 | 42.1 | 47.7 | 30.8 | 65.4 | 26.8 | 26.4 |
| Self-Improving Pretraining (2 rollouts) | 67.1 | 75.2 | 43.8 | 49.9 | 34.3 | 69.0 | 29.0 | 26.7 |
| Self-Improving Pretraining (4 rollouts) | 69.5 | 75.6 | 44.1 | 50.3 | 34.6 | 69.4 | 28.0 | 27.8 |
| Self-Improving Pretraining (8 rollouts) | 70.9 | 75.6 | 45.9 | 51.4 | 35.3 | 71.2 | 30.2 | 28.3 |
| Self-Improving Pretraining (16 rollouts) | 69.1 | 75.8 | 46.1 | 51.7 | 35.7 | 69.4 | 30.0 | 28.3 |
| Pretraining for Factuality | ||||||||
| Pretrain Baseline | 59.6 | 74.2 | 42.2 | 47.7 | 31.3 | 65.3 | 27.0 | 26.7 |
| Self-Improving Pretraining (2 rollouts) | 67.3 | 75.7 | 43.4 | 49.3 | 34.0 | 67.7 | 28.0 | 26.8 |
| Self-Improving Pretraining (4 rollouts) | 68.2 | 76.1 | 44.0 | 50.0 | 34.9 | 68.8 | 28.4 | 27.5 |
| Self-Improving Pretraining (8 rollouts) | 68.3 | 75.6 | 45.8 | 50.8 | 35.7 | 69.6 | 28.6 | 28.2 |
| Self-Improving Pretraining (16 rollouts) | 70.3 | 75.1 | 46.8 | 51.1 | 35.1 | 69.1 | 29.0 | 27.9 |