The suffix judge is used to provide rewards for online RL by scoring the provided completions. In our experiments we consider both online DPO (Qi et al., 2024; Lanchantin et al., 2025) and reward-filtered negative log-likelihood training (RF-NLL) (Christiano et al., 2017), but other update algorithms are possible. Online DPO has shown performance comparable to GRPO (Lanchantin et al., 2025). Unlike GRPO, however, DPO is an off-policy algorithm that allows learning from sequences not generated from the current policy, such as the original suffix or rewrites, making it suitable for our approach. For online DPO we take the chosen completion as the highest scoring, and the rejected as the lowest scoring. For RF-NLL we simply take the highest scoring to conduct an NLL update.

At the beginning of training, we expect the rollouts from the policy to be low quality. Hence, the original suffix and rewrite are most important at this stage. We thus expect that rewarding rollouts should be introduced after sufficient examples have already been seen. Using a rewriter, however, can improve training starting from the initial updates. In our experiments we consider various ablations of including candidate completions of type (i) original, (ii) rewrite and (iii) rollouts, as well as the number of rollouts K.

3 Experiments

3.1 Models and data

Models. We primarily use the pretrained Llama2 1.4 billion parameter model as a baseline policy model (Touvron et al., 2023), and conduct continual sequence pretraining from that checkpoint. Additionally, we conduct pretraining experiments where we train the same model from scratch by first re-initializing the weights. For the sequence pretraining task, we use chunk size N = 128. Both suffix judge and rewriter need to have strong instruction-following capabilities, as such we compare two models: (1) fine-tuned Llama3.1-8B-Instruct (Dubey et al., 2024); and (2) prompted GPT-OSS-120B (OpenAI, 2025).

Data. We use the SlimPajama (SP, Soboleva et al. (2023)) and RedPajama pretraining datasets (RP, Weber et al. (2024)). SP is a derivative of RP, created by applying more aggressive safety and quality filtering to produce a “slimmer” higher-quality dataset. Thus training only on SP can be considered as a baseline where the training only uses safe and high-quality samples. We use RP for training our method in the safety experiments. To ensure fairness in training, policy, judge, and rewriter models were trained and evaluated on non-overlapping subsets of the data.

Judge training. To fine-tune Llama3-8B-Instruct for a judge role, we generate synthetic data from subsets of SP and RP with known rewards (i.e., safe vs. unsafe completions and higher vs. lower quality completions). For the quality task, we create the data by asking a Llama3.3-70B-Instruct (Dubey et al., 2024) model to spoil the original suffix (see Appendix Figure 10) extracted from SP. A pair of original and corrupted suffixes is then used to create two samples, by wrapping the pair or its flipped version in the quality judge prompt, given in Figure 3. For the safety task, we use the same model to filter safe and unsafe suffixes from RP by prompting it (see Figure 2) to evaluate the safety of the suffix with 8 random seeds, and only using samples where all 8 judgments are safe or unsafe. We then use the same prompt to wrap suffixes for training. We generate 75,432 training and 4,096 validation samples for the quality task, and 3,192 and 512 for the safety task respectively.

Rewriter training. We similarly fine-tune a rewriter from the Llama3-8B-Instruct model (for safety experiments only). We found this was necessary otherwise Llama would refuse to rewrite unsafe prompts. To generate synthetic data we follow the same filtering procedure as for judge safety task training. 73,080 safe and unsafe suffixes then used with the rewriter prompt template, provided in Appendix Figure 11.

Policy training. Training on SP is conducted on 983,520 samples. Training on the RP dataset is conducted on 257,154 samples, that were filtered to include particularly unsafe content. Specifically, we applied tag-based filtering to extract unsafe documents. Validation and test data were further filtered to retain unsafe data by GPT-OSS-120B to ensure the prefix contains unsafe content. Filtering details are provided in Appendix B.1.