At inference time to make a judgment we query the model twice, for safety using Figure 2, and for quality using Figure 3, and combine the results. For the safety judgment, this is a pointwise score, but for quality this is a pairwise judgment given two candidate responses, which outputs which is better of the two. For the latter during policy training we run all pairwise comparisons amongst candidates in the batch, assigning reward 0 or 1 in each case, and take the average of their rewards to obtain pointwise scores. For each rollout, our judge is prompted to evaluate safety and quality 5 times each with temperature T = 1.0 and top_p = 0.6.

Suffix rewriter. Similarly, we train Llama3.1-8B-Instruct model with the GRPO algorithm. Our goal is to build a suffix rewriter that leaves safe high quality suffixes unchanged (hence the generative output would typically copy the suffix that is given in the input context), whereas for unsafe suffixes, they should be rewritten to be safe. Hence, to train the rewriter, the reward is assigned with the following method:

• If the model was prompted to rewrite a safe suffix, we return reward 1.0 if the rewritten suffix x_j is an exact match of the given suffix x_j, otherwise we reward it with 0.0:
  R_safe = {

  1.0 if x_j = x_j ,

  0.0 otherwise .

  (1)
• If the model was prompted to rewrite an unsafe suffix, the rewritten suffix is evaluated with the suffix judge based on quality J_qual and safety J_safe, averaging judgments across 5 random seeds:
  R_unsafe =

  1

  ---

  2

  (J_qual(x_j, x_j|x_1,...,j−1) + J_safe(x_j)) .

  (2)

To train the suffix rewriter model we use same setup as for the suffix judge. We modify the maximum prompt length to 3968 tokens, and the model generation length to 128 new tokens to match our suffix length. We validate model performance on safe and unsafe subsets. We observe steady improvement on the copy task (exact match reward score on safe suffixes, Figure 6), and use the final checkpoint that achieves token overlap percent plateaued at 98%, as shown in Figure 7.

3.2.2 Factuality Experimental Setup

Suffix judge. In the factuality training setting, we only consider using a judge, and not a rewriter. For the factuality judge, this is a pointwise judgment given one candidate response, and a reference answer. We use the original suffix from the training data as the reference. We use GPT-OSS-120B with the prompt given in Figure 4. In subsection A.2, we conduct a detailed study using different strong post-trained models as the judge, and various prompt designs, comparing their performance.

The suffix judge outputs whether the continuation has no hallucination (reward 1), possible hallucination (reward 0.5) or definite hallucination (reward 0). As in the safety experiments, we combine this reward with an overall quality score of the generation, by adding the quality scoring judge rewards. This is done in the same way as in subsubsection 3.2.1. GPT-OSS-120B is prompted with temperature T = 1.0 and top_p = 1.0.