Figure 6 Suffix rewriter validation rewards on safe and unsafe suffixes of the RedPajama dataset. Initial performance of the model is close to random chance on the safety task (0.5 score on unsafe suffixes), and near zero on copying safe suffixes (exact match reward score of 0.1), but still increasing after 500 steps.
Figure 7 Token Overlap in Suffix Rewriter Validation on RedPajama Dataset. We evaluate token overlap between original and rewritten suffixes for both safe and unsafe suffixes in the RedPajama dataset. Our objective is to produce safe rewrites that remain similar to the original suffix. Token overlap serves as a measure of this similarity. For safe suffixes, token overlap increases and approaches 1.0 as we optimize for exact matches. In contrast, token overlap for unsafe suffixes averages around 0.63 and remains close to its initial value, indicating less change (should not overlap).
We also consider another variant of using a single pivot candidate for pairwise comparisons instead, resulting in K judgments for each update, rather than (K2).
3.2.3 Quality Experimental Setup
Suffix judge. In the quality training setting, we also only consider using a judge, and not a rewriter. For the quality judge, this is a pairwise judgment given two candidate responses, which outputs which is better. For this we use GPT-OSS-120B with the prompt given in Figure 3. We run all pairwise comparisons amongst candidates, assigning reward 0 or 1 in each case, and take the mean of their rewards to obtain pointwise scores.
We also consider two other variants: (1) using the trained model from subsubsection 3.2.1 but only prompted for quality; and (2) using a single pivot candidate for pairwise comparisons instead, resulting in K judgments for each update, rather than (K2).
3.2.4 Policy training variants and ablations
We conduct a series of variants and ablations of policy training primarily in the safety pretraining setting. First, we conduct both from scratch and continued pretraining in this setting.
Continual pretraining experiments Self-Improving Pretraining models are trained with online DPO (unless said otherwise in ablations) with the global batch size 256, sampling 16 rollouts per prompt using temperature T = 1.0 and top_p = 1.0. We train on 64 GPUs for 2000 steps with cosine learning rate lr = 5.0e −06, min ratio 0.1, and 100 warmup steps. The maximum sequence length is set to 2048 tokens, and the model