How it works
Code counts. The model reads.
The rule
Wherever an answer can be counted (dates, citation counts, how many papers match a search, intervals), code does it. The model is kept for reading and judgement. SPEC.md:81 The code never calls a model. SPEC.md:56
- No quote, no claim. Every claim a model makes about a paper needs an exact quote, and code must find that quote in the stored text. The model never supplies IDs, years or citation counts; code looks them up. SPEC.md:344
- The checker is not the reader. A separate agent, with no stake in a claim, searches for work that already answers it. SPEC.md:347
- Every search is logged. A report says how many ways it searched and what the closest work was, never "nobody studies this". SPEC.md:113
- Retractions are checked for every paper used as evidence. SPEC.md:348
The tool
The pipeline that sweeps a topic for open questions, its guardrails and how it is tested are described on The tool. This page covers how the studies on this site were run and which models did what.
A study, step by step
The results on this site come from studies built on the same parts. A plan is written and committed before any reading. AI readers code each paper against a fixed codebook and give a quote for every code. Code checks the quotes and computes every count and interval. Where a check by another model or a person exists, the page says so in its status box.
AI-use statement
Every model used for the results on this site, and what it did.
| Model | Job |
|---|---|
| Claude Sonnet (Anthropic) | Main reader of PCOS trial reports in the outcomes count; second-reader monitor in the three-condition count; both readers in the methods study; both readers for the priorities map; the agents that rate abstracts, pull out gaps, search for prior work and score. experiments/outcome_audit/CROSS_CONDITION_RESULTS.md:260,273 experiments/measurand_tiers/repass/REPORT_v3.md:40 experiments/mismatch_map/pmos/PREREG.md:20 .claude/agents/*.md |
| Claude Haiku (Anthropic) | Reader for part of the PCOS trial reports; main reader for endometriosis and male infertility. docs/REVIEW_PMOS_AUDIT_2026-10-26.md:175 experiments/outcome_audit/CROSS_CONDITION_RESULTS.md:259 |
| Claude Opus 5.5 (Anthropic) | Reader of companion papers in the outcomes count; independent reviews and red-team reviews; drafted the pages of this site. experiments/outcome_audit/LINKAGE_RESULTS.md:151 docs/REVIEW_PMOS_AUDIT_2026-10-26.md:3 |
| Claude Fable (Anthropic) | One benchmark of outcome coding only. experiments/outcome_audit/BENCH.md:11 |
| Gemini 4 argon (Google) | Independent re-read of 169 outcome-count records. experiments/outcome_audit/crosscheck/RESULTS_full.md:37-42 |
| DeepSeek v4.1 flash (DeepSeek) | Independent re-read of the same 169 records. experiments/outcome_audit/crosscheck/RESULTS_full.md:5-10 |
| No model | All counts, intervals and tests; the gut gene counts (sequence alignment by code). |
Claude drafts the prose on this site from the results files, and Dan Bonomo edits it. Each page says at the foot whether his edit is done. Formal papers with a DOI are written by Dan.
What it cost
- Re-reading 169 records with the two other vendors' models: $2.995 (Gemini) and $0.181 (DeepSeek). experiments/outcome_audit/crosscheck/RESULTS_full.md:10,42
- The main reading of the methods study: 6.62M tokens. experiments/measurand_tiers/repass/REPORT_v3.md:150
- One known-answer test question, small run: 1.69M tokens on average. bench/SCORECARD.md:16
Model work for the Claude readers runs under a subscription, so most costs are counted in tokens rather than dollars.