Figure 4 Factuality judge prompt.
Factuality judge prompt
You are a careful evaluator of factual accuracy.
You will be given an original text, a human continuation (assumed accurate ground truth), and a model continuation. Your goal is to check for factual inaccuracies or clearly impossible statements in the model continuation, not to judge style, narrative flow, or coherence.
Important:
- The original text may be incomplete; treat it as partial ground truth.
- The human continuation should be treated as the primary ground-truth reference (assumed accurate),
but you may also use your own general world knowledge as valid ground truth. - Factual statements are considered correct if they are supported by the original text, the human continua-
tion, or by reliable general world knowledge. - Only statements that are clearly false, self-contradictory, or impossible should count as hallucinations.
- Narrative oddities, off-topic content, or unusual story events should not count as hallucinations if
they are plausible or creatively reasonable. - Minor invented terms, creative liberties, or loosely grounded content should be treated as Possible Hallucination rather than Definite Hallucination, unless they contradict the human continuation
or known facts. - Do not penalize the model for minor semantic or logical quirks that occur naturally in story continuations.
Original Text: {original_text}
Human Continuation (ground truth): {human_continuation}
Model Continuation: {model_output}
Please think step by step:
- Use the human continuation and the original text as your primary references. Also allow your general world knowledge where relevant.
- Identify any statements in the model continuation that are factually incorrect, clearly impossible, or that directly contradict the human continuation or the original text.
- Ignore content that is off-topic, loosely connected, creative, or unusual but plausible.
- Summarize your reasoning, then assign one label:
- No Hallucination — model continuation is consistent with known facts and the human continu-ation (any differences are plausible creative choices).
- Possible Hallucination — minor, creative, or uncertain content that may be inaccurate but is not definitively false or contradictory to the human continuation.
- Definite Hallucination — contains clear factual errors, impossible claims, or direct contradictions with the human continuation or known facts.
Respond in JSON format:
{
"reasoning": "your reasoning here",
"label": "No Hallucination" | "Possible Hallucination" | "Definite Hallucination"
}
6