generates N = 128 new tokens for each rollout. For the safety task, the fine-tuned Llama3.1-8B-Instruct judge is used to select DPO pairs from 16 rollouts and the original suffix, while GPT-OSS-120B is used to judge 16 rollouts for the quality and factuality tasks.

From-scratch pretraining To pretrain from scratch, we use a similar setup, but increase the number of training steps to 21,000, increase the learning rate to 5.0e −04, and the number of warmup steps to 2000. In these experiments, we only use 1 rollout for training.

Ablations We also ablate various ways of doing the training with different loss functions and candidate generation pools during online training, all compared to next token prediction baselines.

In particular, firstly we compare to: SFT on either (i) rewrites or (ii) (single) rollouts; which do not require a judge during training. For RL training, we use online DPO, which has shown performance comparable to GRPO (Lanchantin et al., 2025). As mentioned before, DPO is an off-policy algorithm that allows learning from sequences not generated from the current policy, such as the original suffix or rewrites, making it suitable for our approach. First, a baseline simple option is to use the rewrite as the chosen and the current rollout as the rejected in online DPO, which also does not require a judge, inspired by the approach in Chen et al. (2024).

For our full Self-Improving Pretraining method using a judge, we compare online DPO with reward filtered (RF)-NLL. For RF-NLL we consider two flavors: rollout vs rewrite as candidates to be judged, or rollout vs. original suffix vs rewrite. For online DPO, we consider: (i) suffix vs 1 rollout, (ii) rewrite vs. 1 rollout, (iii) suffix vs. 16 rollouts; and (iv) 16 rollouts only. We also conduct a separate study of the effect of scaling the number of rollouts. For policy model generations during training we use a temperature of 1.

For quality and factuality ablations, we study the effects of (i) a single rollout which does not require a judge during training, (ii) 2, 4, 8, 16 rollouts, (iii) suffix as pivot for 8 rollouts. We also compare using the trained judge from subsubsection 3.2.1, with GPT-OSS-120B as an online judge in the quality pretraining setting.

3.3 Evaluations

We evaluate our models on a broad set of benchmarks, including standard evaluations and additional benchmarks focused on coherence, safety and factuality. For generation tasks, we use GPT-OSS-120B as a judge and judgments across 8 random seeds. For the policy model we use greedy generations.

Generation quality. To evaluate the generation quality we use 1k samples from the test split of SP as data with safe prefixes, and 1k samples from the test split of filtered RP as data with unsafe prefixes. Generation quality is evaluated by comparing a sequence of length N against baseline generations of Llama Base of the same length. We use GPT-OSS-120B as a suffix judge using the prompt given in Figure 3. We average judgments across 8 random seeds using a temperature of 0.7. In addition, we measure coherence, particularly in terms of repetition, independently using the prompt given in Figure 13. Note that the generation quality score (win rate) is hence 50.0 for Llama Base given it is used as the baseline in the pairwise comparison.

Standard Evaluations. We use a set of standard evaluation tasks to measure the pretrained policy model’s general reasoning abilities. In particular, we average performance across the following datasets: BoolQ (Clark et al., 2019), PIQA (Bisk et al., 2020), SIQA (Sap et al., 2019), HellaSwag (Zellers et al., 2019), ARC easy and challenge (Clark et al., 2018), OpenBookQA (Mihaylov et al., 2018), and 5-shot performance on the aggregated MMLU benchmark (Hendrycks et al., 2020).

Safety. The policy model’s safety is evaluated as a weighted average across five datasets: the RP test split, RealToxicityPrompts (Gehman et al., 2020), ToxiGen (Hartvigsen et al., 2022), and the XStest safe and unsafe sets (Röttger et al., 2024). In each case, safety is evaluated with GPT-OSS-120B as a judge using the prompt given in Figure 2. We use majority vote over N predictions with a temperature of 1.

Factuality. The policy model’s factuality is evaluated as a weighted average across five datasets: the RP test split, FActScore (Min et al., 2023), HaluEval (Li et al., 2023), which are generation tasks, and the TruthfulQA multiple-choice tasks MC1 and MC2 (Lin et al., 2022). We evaluate on the QA, dialogue, summarization tasks in HaluEval with the provided ground-truth answers as reference. For FActScore, the provided wikipedia text