We report ablation results on the number of rollouts used in online DPO for quality, factuality, and safety training in Figure 9. We generally find improved performance across all benchmarks with an increasing number of rollouts, where we experimented with between 1 and 16 rollouts. We did not experiment past 16 rollouts due to the increased compute required, but we expect further gains.

Furthermore, similar trends can be seen in generation quality and standard evaluations, as shown in Appendix Table 14, where more rollouts lead to better final performance across all benchmarks tested. Detailed standard task results are also given in Appendix Table 15 and Table 16.

As mentioned in subsection 3.1, we experiment with two types of judges: one fine-tuned specifically for a target task such as quality, and another used directly via prompting without training. In Table 7 we compare these two judges when they are used for quality training. We find that the prompted GPT-OSS-120B model generally performs better, but the finetuned Llama judge is not far behind, demonstrating that we can purpose-train a smaller model for this goal. A detailed breakdown of results across standard tasks can be found in Appendix Table 17.

We also experiment with speeding up pairwise quality judgments by instead using a pivot. That is, one generation is selected and then all generations in the training batch are compared only against this pivot.