Table 10 Prompting GPT-4o for factuality predictions on 200 examples in the SlimPajama test set.
| Continuation evaluated | Prompt | Definite | Possible | No Hallucination |
|---|---|---|---|---|
| Original Suffix | v1 (No-Ref) | 79 (39.5%) | 36 (18.0%) | 85 (42.5%) |
| Original Suffix | v2 (No-Ref) | 15 (7.5%) | 27 (13.6%) | 157 (78.9%) |
| Original Suffix | v3 (No-Ref) | 3 (1.5%) | 54 (27.0%) | 143 (71.5%) |
| Original Suffix | v4 (No-Ref) | 4 (2.0%) | 60 (30.0%) | 136 (68.0%) |
| Model | v1 (No-Ref) | 180 (90.5%) | 9 (4.5%) | 10 (5.0%) |
| Model | v2 (No-Ref) | 142 (71.0%) | 21 (10.5%) | 37 (18.5%) |
| Model | v3 (No-Ref) | 104 (52.0%) | 70 (35.0%) | 26 (13.0%) |
| Model | v4 (No-Ref) | 86 (43.4%) | 76 (38.4%) | 36 (18.2%) |
| Model | v1 (With-Ref) | 185 (94.4%) | 2 (1.0%) | 9 (4.6%) |
| Model | v2 (With-Ref) | 153 (77.3%) | 7 (3.5%) | 38 (19.2%) |
| Model | v3 (With-Ref) | 143 (71.5%) | 26 (13.0%) | 31 (15.5%) |
| Model | v4 (With-Ref) | 125 (62.5%) | 39 (19.5%) | 36 (18.0%) |
| Model | v5 (With-Ref) | 87 (44.2%) | 19 (9.6%) | 91 (46.2%) |
Table 11 Prompting GPT-OSS-120B for factuality predictions on 200 examples in the SlimPajama test set.
| Continuation evaluated | Prompt | Definite | Possible | No Hallucination |
|---|---|---|---|---|
| Original Suffix | v1 (No-Ref) | 106 (53.0%) | 30 (15.0%) | 60 (30.0%) |
| Original Suffix | v2 (No-Ref) | 63 (31.5%) | 28 (14.0%) | 108 (54.0%) |
| Original Suffix | v3 (No-Ref) | 66 (33.0%) | 56 (28.0%) | 77 (38.5%) |
| Original Suffix | v4 (No-Ref) | 42 (21.0%) | 54 (27.0%) | 101 (50.5%) |
| Model | v1 (No-Ref) | 159 (79.5%) | 14 (7.0%) | 25 (12.5%) |
| Model | v2 (No-Ref) | 133 (66.5%) | 11 (5.5%) | 56 (28.0%) |
| Model | v3 (No-Ref) | 112 (56.0%) | 46 (23.0%) | 41 (20.5%) |
| Model | v4 (No-Ref) | 106 (53.0%) | 42 (21.0%) | 52 (26.0%) |
| Model | v1 (With-Ref) | 185 (93.43%) | 2 (1.01%) | 11 (5.56%) |
| Model | v2 (With-Ref) | 159 (79.5%) | 14 (7.0%) | 27 (13.5%) |
| Model | v3 (With-Ref) | 131 (65.83%) | 46 (23.12%) | 22 (11.06%) |
| Model | v4 (With-Ref) | 102 (54.84%) | 62 (33.33%) | 22 (11.83%) |
| Model | v5 (With-Ref) | 82 (43.39%) | 28 (14.81%) | 79 (41.80%) |
Table 12 Prompting Llama3-70B for factuality predictions on 200 examples in the SlimPajama test set.
| Continuation evaluated | Prompt | Definite | Possible | No Hallucination |
|---|---|---|---|---|
| Original Suffix | v1 (No-Ref) | 21 (10.8%) | 22 (11.3%) | 152 (77.9%) |
| Original Suffix | v2 (No-Ref) | 1 (0.5%) | 0 (0.0%) | 199 (99.5%) |
| Original Suffix | v3 (No-Ref) | 1 (0.5%) | 9 (4.5%) | 189 (95.0%) |
| Original Suffix | v4 (No-Ref) | 0 (0.0%) | 3 (1.5%) | 195 (98.5%) |
| Model | v1 (No-Ref) | 74 (38.3%) | 62 (32.1%) | 57 (29.5%) |
| Model | v2 (No-Ref) | 29 (14.5%) | 9 (4.5%) | 162 (81.0%) |
| Model | v3 (No-Ref) | 20 (10.0%) | 90 (45.0%) | 90 (45.0%) |
| Model | v4 (No-Ref) | 19 (9.5%) | 63 (31.5%) | 118 (59.0%) |
| Model | v1 (With-Ref) | 160 (81.6%) | 29 (14.8%) | 7 (3.6%) |
| Model | v2 (With-Ref) | 91 (45.5%) | 46 (23.0%) | 63 (31.5%) |
| Model | v3 (With-Ref) | 41 (20.5%) | 131 (65.5%) | 28 (14.0%) |
| Model | v4 (With-Ref) | 49 (24.7%) | 109 (55.1%) | 40 (20.2%) |
| Model | v5 (With-Ref) | 31 (15.5%) | 37 (18.5%) | 132 (66.0%) |
Table 13 Factuality prompt agreement metrics: Llama3.3-70B and GPT-OSS vs. GPT-4o.
| Reference Set | Llama3.3-70B vs. GPT-4o | GPT-OSS vs. GPT-4o |
|---|---|---|
| Orig. Suffix No-Ref | Agreement: 63.64% – 79.68% (avg: 71.33%) Gap: 2.70% (∼3.8% relative) |
Agreement: 47.24% – 61.11% (avg: 54.82%) Gap: 10.90% (∼19.9% relative) |
| Model No-Ref | Agreement: 32.93% – 50.00% (avg: 37.47%) Gap: 21.23% (∼56.7% relative) |
Agreement: 65.66% – 85.28% (avg: 72.32%) Gap: 12.56% (∼17.4% relative) |
| Model With-Ref | Agreement: 38.61% – 84.62% (avg: 54.77%) Gap: 24.72% (∼45.1% relative) |
Agreement: 56.99% – 94.85% (avg: 73.21%) Gap: 21.10% (∼27.8% relative) |