I will provide you with a text. Please tell me if the text is safe or not.
First, provide your reasoning. Please be brief. Then, conclude your brief reasoning with "FINAL DECISION: YES" or "FINAL DECISION: NO". You should only say yes or no.
Text: {suffix}
I have an original text fragment and 2 generated continuations that follow the original text fragment.
Original text fragment:
{text}
Option 1: {continuation 1}
Option 2: {continuation 2}
Judge which continuation is more coherent. If both continuations are coherent, choose the one that is more factual and otherwise high quality. Ignore whether continuations feel incomplete. In the final paragraph, output "Conclusion: Option 1" if you think option 1 is better otherwise output "Conclusion: Option 2".
3.2 Experimental Setup
3.2.1 Safety Experimental Setup
Our pipeline involves the following three models: a judge, rewriter, and the policy model. Below we will summarize the setup for the components.
Suffix judge. Recent studies provide strong evidence that LLM judges become more robust and effective when they generate their own Chain-of-Thought (CoT) analyses before producing final judgments (Zhang et al., 2024a; Chen et al., 2025c; Whitehouse et al., 2025). To fine-tune Llama3.1-8B-Instruct to be a safety and quality judge incorporating reasoning, we use GRPO (Shao et al., 2024) as our optimization algorithm. Unlike SFT, GRPO does not require generating high-quality synthetic CoT data, but fully relies on a signal from the final judgment, while incentivizing reasoning traces that result in correct judgments. To reward the judge model during training we rely on labels from synthetically generated data of judgments, rewarding a correctly categorized suffix with 1.0, and 0.0 for mismatching the label.
We run GRPO training on the synthetically generated data, where the judge is simultaneously trained on two tasks: quality and safety. We set the global batch size to 256, with 16 generations per prompt under temperature T = 0.6 and top_p = 0.6. We train on 64 GPUs for 500 steps with 2.0e −07 constant learning rate. The maximum prompt length is set to 3584 tokens, and the model can generate up to 512 new tokens.
During training we observe that initially the safety task is easier to learn than quality, as the model plateaus at 0.94 average reward score at approximately 100 steps, while the reward for quality keeps growing until the end of the training, see Figure 5. Manually analyzing judgments, we found that the initial model tends to favor suffixes that feel more complete, rather than those that are more coherent with respect to the context.