What the machine gets wrong
Measured failures first, then the rules our own agents broke.
1. It rarely notices that a question has already been answered
We gave the machine four questions whose answers were already in print before a cut-off year. It should find the answering paper and close the gap, or narrow it with that paper named. bench/SCORECARD.md:7
| Reading | Result |
|---|---|
| Headline: every gap from the papers that raised the question was closed, or narrowed with the answering paper named | 1 of 4 |
| Strict: every matching gap, including neighbouring questions | 0 of 4 |
| Best matching gap closed or narrowed on the answering paper | 4 of 4 |
Across the four questions, 9 of 20 matching gaps were still left open when the answer was in print. palace/threads/challenge-closes-nothing.md:24 In the PCOS sweep, the search for prior work closed 1 of 123 gaps. palace/threads/challenge-closes-nothing.md:37
So the machine can find the paper that answers a question, but the gaps next to it often stay open. Any gap it reports may already be answered. That is why every gap carries the searches that were run and the closest work found.
2. The same question can come out differently on different runs
One test question went closed, narrowed, closed, narrowed across four versions of the code. bench/SCORECARD.md:45-49 docs/aar/AAR_2026-11-07_r-2026-11-07-1.md:29-30
3. "Nobody has studied this" is usually too strong
In a fact-check of an AI's claims about sickle cell trait, every "nobody has studied X" claim was softer than stated: 6 of 6. palace/threads/challenge-closes-nothing.md:38
4. Readers miss things, and two copies of one model miss the same things
- When the pilot reading of how papers state their methods was spot-checked, Dan agreed with the readers on 17 of 20 rows. All three misses were the readers saying "stated" where he said "not stated", and both readers made each miss. experiments/measurand_tiers/repass/REPORT_v3.md:46-47,123
- In the outcomes count, both Claude Sonnet readers missed prediabetes returning to normal in one trial; three other models coded it. experiments/outcome_audit/LINKAGE_RESULTS.md:108-111
- Models from two other vendors agreed with Claude's headline codes on 160 and 147 of 169 records. Where they differ, some misses are likely Claude's and some theirs; a person has not yet judged them. experiments/outcome_audit/crosscheck/RESULTS_full.md:7,39
5. Code errors found by review
- Linking papers to trials joined every record that shared any registry ID. One registry overview listing 15 IDs chained 8 endometriosis trials into one. A reviewer caught it; the counts were corrected and the first figures kept. experiments/outcome_audit/CROSS_CONDITION_RESULTS.md:203-208
- 17 of the 602 papers in the outcomes count have a linked retraction. They are still in the count. experiments/source_check/outcome-audit/SOURCE_CHECK.md:15
6. Facts supplied from memory
- A paper ID supplied from memory was wrong; a logged search caught it. palace/threads/gut-androgen.md:43
- A study's headline organism was first described from memory as reduced in PCOS. The abstract says it was elevated. palace/threads/gut-androgen.md:76
This is why the rule is that a model never supplies an ID, a year or a count. Code looks them up.
7. Rules our agents broke
Every rule that lived only in an agent's instructions has been broken at least once. palace/threads/guardrails.md:7
- Agents building test questions opened answer files they were not meant to see. docs/aar/AAR_2026-11-01_bench_night.md:42
- Agents running a test stopped early twice, leaving work half done. docs/aar/AAR_2026-11-01_bench_night.md:38
- A test copied files into a temporary folder and filled the disk. docs/aar/AAR_2026-11-01_bench_night.md:39
The fix is to move rules out of instructions and into code that refuses the action, then test that the refusal works.
8. Cost estimates run over
- One test question cost 2.2 times its estimate; another cost 5.24M tokens against a 3.7M estimate. docs/aar/AAR_2026-11-01_bench_night.md:44
- A sample reading cost about 0.96M tokens against 0.4M estimated. docs/DEVLOG.md:649-650
- A re-read cost 0.80M tokens against 0.43M expected. experiments/measurand_tiers/repass/REPORT_v3.md:148