| Aspect | v1 | v2 | v3 | v4 | v5 |
|---|---|---|---|---|---|
| Main Focus | Factuality + logical coherence | Factual accuracy only | Factual accuracy with creative tolerance | Factual accuracy, style-agnostic | Clear factual inaccuracies only |
| Role Description | “Factuality evaluator” | “Careful evaluator of factual accuracy” | “Careful evaluator of factual accuracy” | “Careful evaluator of factual accuracy” | “Careful and forgiving evaluator” |
| Treatment of Original Text | Must logically follow from original + human continuation | May be incomplete; partial ground truth | May be incomplete; partial ground truth | May be incomplete; partial ground truth | May be incomplete; partial ground truth |
| World Knowledge Use | Implicit (through human continuation) | Explicit: “use your own general world knowledge” | Explicit: “use your own general world knowledge” | Explicit: “reliable general world knowledge” | Explicit: “rely on general world knowledge” |
| Coherence Requirements | Strict: must logically follow | Relaxed: “Minor logical or coherence issues should NOT be considered hallucinations” | Ignored: “Minor coherence issues should NOT automatically count” | Ignored: “not to judge style, narrative flow, or coherence” | Ignored: “Ignore coherence gaps, logical quirks” |
| Handling of Off-topic Content | Penalized as implausible | Ignored if factually accurate | “should NOT automatically count as hallucinations” | “should NOT count as hallucinations if plausible” | “should NOT be penalized” |
| Creative/Invented Terms | Penalized if implausible | Allowed if not false/contradictory | “Possible Hallucination rather than Definite” | “Possible Hallucination rather than Definite” | “should be treated as Possible Hallucination at most” |
| Narrative Oddities | Not explicitly addressed | Not explicitly addressed | Not explicitly addressed | Explicitly allowed: “should NOT count as hallucinations if plausible” | Explicitly allowed: “should NOT be penalized” |
| Obscure/Unverifiable Entities | Not explicitly addressed | Not explicitly addressed | Explicitly allowed: “should NOT automatically count as hallucinations” | Implicitly allowed through style tolerance | Explicitly allowed: “should NOT be penalized” |
| Semantic/Logical Quirks | Likely penalized | Not explicitly addressed | Not explicitly addressed | Explicitly allowed: “Do not penalize minor semantic or logical quirks” | Explicitly allowed: “Ignore logical quirks” |
| Goal | Determine hallucinations, inconsistencies, implausibilities | Check factual inaccuracies | Check factual inaccuracies or clearly implausible statements | Check factual inaccuracies or clearly impossible statements (not style) | Check clear factual inaccuracies or impossible statements (not style/coherence) |
| Strictness Level | Strictest | Moderate | Relaxed | Very Relaxed | Most Lenient |
| Error Threshold | “hallucinations, internal inconsistencies, or statements implausible” | “false, self-contradictory, or implausible” | “clearly false, self-contradictory, or impossible” | “clearly false, self-contradictory, or impossible” | “clearly and unambiguously false, self-contradictory, or impossible” |
| Uncertainty Handling | Standard evaluation | Standard evaluation | Standard evaluation | Standard evaluation | “If uncertain whether something is false, lean toward No Hallucination” |
Table 9 Comparison of v1–v5 with reference Prompt Variants. Versions v1 and v2 are overly strict, flagging too much content for rewriting and risking overly generic rewrites by removing valid creative elements. Versions v3 and v4 allow more creativity, with v4 explicitly excluding style and narrative from the evaluation scope. Version v5 is potentially too lenient, as it biases toward “No Hallucination”, may miss subtle factual errors and fails to catch issues that could make rewritten text inaccurate. We hence use v4 in our main experiments.
C Evaluation results
C.1 Evaluation prompts
The judge prompts for coherence, FActScore, HaluEval can be found in Figure 13, Figure 14, and Figure 15, respectively.