A Additional Judge experiments

A.1 Suffix judge comparisons for quality

We compared several medium-sized post-trained models on the quality task we use for judge training. We used our synthetic data from the SP validation subset and asked Llama3.1-8B-Instruct, Llama3.3-70B-Instruct, DeepSeek-R1-Distill-Llama-8B, and DeepSeek-R1-Distill-Llama-70B. Results are summarized in Table 8. We found that all models underperform on this task and thus cannot be used as Judge without further fine-tuning. The main problem we found is that the models tend to favor suffixes that feel more complete, rather than those more coherent with respect to the context. Training helps to fix this problem.

Table 8 Accuracy on the synthetic binary choice quality task used for judge training. We measure different families and sizes of judge on subsets of the SlimPajama dataset.

Quality Judgments	arxiv	book	c4	github	stack_exch	wiki	cc
Llama3.1-8B-Instruct	48.4	45.0	54.9	55.1	51.9	55.9	53.0
Llama3.3-70B-Instruct	72.7	84.0	75.5	69.9	66.0	84.8	74.1
DeepSeek-R1-Distill-Llama-8B	57.8	55.3	57.5	55.8	53.2	63.5	56.8
DeepSeek-R1-Distill-Llama-70B	73.6	79.4	75.4	72.6	64.0	86.0	74.9

Figure 10 Synthetic data generation corruption prompt used to create negative data for suffix judge training.

Synthetic data generation corruption prompt

Your task is to modify a continuation of a given context. Context: {prefix} Continuation: {suffix} Your continuation must closely resemble provided continuation, but it should contain some factual inaccuracies or unsafe content. IMPORTANT: Please strictly follow the following format: MODIFIED CONTINUATION: <provide only modified continuation here, without any additional text or explanations>

A.2 Suffix judge comparisons for factuality

To evaluate different strong post-trained models as judges to measure factuality, here we conduct experiments by prompting GPT-4o, GPT-OSS-120B, and Llama3.1-70B-instruct with a test set of 200 SlimPajama instances. We try 5 versions (v1-v5) of a so called “with-reference” prompt with the (typically human-written) original suffix used as a reference to judge the factuality of a model completion given the prefix, and 4 versions (v1-v4) of a so-called “no reference” prompt, where the original suffix is not given as reference. The prompts are described in subsubsection A.2.1 and subsubsection A.2.2. A summary of different versions of the with-reference prompts we tried according to various aspects can also be found in Table 9.

We present evaluation results in Tables 10, 11 and 12. Overall, we find that by providing the original (typically human-written) suffix as a reference, the post-trained models perform better at the factuality judgment task when judging model generations. Through manual annotation, we find GPT-4o tends to provide the best evaluation results. Then, considering GPT-4o’s prediction as a reference label, we calculate the agreement ratio between GPT-OSS-120B or Llama3.1-70B-instruct with GPT-4o. The results are given in Table 13. We find that combining both our manual inspection and the overall metrics results, that GPT-OSS-120B performs better as a factuality judge than Llama3.1-70B-instruct, and is thus used in subsequent experiments with the v4 with-reference prompt.