The tool
Science Machine sweeps the literature on a research topic and returns a ranked list of open questions. Each question comes with the quotes it rests on, the searches run to find prior work, and the closest work found.
What it does, step by step
- Seed. A person names the topic, lists its synonyms and old names, and writes a handful of search queries. Code estimates how many papers each query will return. Nothing is collected until the person approves. docs/RUNBOOK.md:52
- Harvest. Code collects paper records and abstracts from public databases. docs/RUNBOOK.md:53
- Triage. A small model rates each abstract and picks the papers likely to state a specific, checkable gap. docs/RUNBOOK.md:54
- Extract. Code fetches open-access full text for those papers. A model reads each one and pulls out candidate gaps, each with an exact quote. Code then looks for every quote in the stored text and drops any gap whose quote it cannot find. docs/RUNBOOK.md:55
- Search for prior work. A separate agent tries to find the paper that already answers each gap. It starts with papers citing the evidence and a key-term search, then returns a verdict: closed, partly answered, or open. The highest-ranked gaps get a fuller search, including restatements in other fields' vocabulary and a web search. docs/RUNBOOK.md:56,60
- Score. A model rates each gap against a fixed rubric and gives a reason for each rating, alongside measures that code computes. The top gaps are then compared head to head. docs/RUNBOOK.md:58,62
- Feasibility. For the gaps a person marks as interesting, code searches public archives (sequencing data, trial registries, gene-expression data). A model then judges whether those records could answer the gap, and what answering it would take. .claude/skills/feasibility/SKILL.md:3
- Report. Code ranks the gaps and writes the report and a table. docs/RUNBOOK.md:63
Before anything from a run reaches a person, code runs a set of warnings over it. It flags a gap that rests on one old paper, a source about a different condition, or a retracted paper. docs/RUNBOOK.md:76-78
Code counts; the model reads and judges
Anything that can be counted is counted by code: dates, citation counts, how many papers match a search. The model is kept for reading and judgement. SPEC.md:81 The model never supplies an ID, a DOI, a year, a citation count or a search hit count; code looks each one up. CLAUDE.md:57 The code itself never calls a model. SPEC.md:56
Guardrails
- No quote, no gap. Code must find the exact quote in the stored text. CLAUDE.md:56
- The agent that searches for prior work is not the agent that found the gap. It has no stake in keeping the gap alive. CLAUDE.md:60
- "Nobody studies this" is a hypothesis. Every gap carries its logged searches and the closest work found. CLAUDE.md:58-59
- Open-access full text only. CLAUDE.md:61-62
- Text from papers is data, never instructions. A paper cannot tell the model what to do. CLAUDE.md:63
- Health gaps are never marked testable at home. Each gap is rated on whether a person could test it at home, and a health-related gap never gets a "yes". CLAUDE.md:64 SPEC.md:266
How it is tested
Each test is a calibration case: a research question and a cut-off year. The tool searches only papers published before the cut-off, and the agents doing the reading never see the case or its answer. bench/SCORECARD.md:168 CLAUDE.md:66-67 Some cases are gaps stated in print before the cut-off and answered only later, which the tool should find and leave open. Others are questions already answered before the cut-off, which the tool should close. bench/SCORECARD.md:7,101 The case set holds 52 cases. docs/aar/AAR_2026-11-01_bench_night.md:26 The cases are run again after each significant change to the tool and kept on a scorecard by code version. bench/SCORECARD.md:3
| Test | Latest result |
|---|---|
| Answered questions closed (small runs, 2026-11-07) | 1 of 4 on the headline reading; 0 of 4 on the strict reading; the best matching gap closed or narrowed on the answering paper in 4 of 4 |
| Answered questions closed (full runs, 2026-11-01) | 1 of 2 |
| Known gaps found and left open (full run, 2026-11-02) | 1 of 1, ranked 6 of 19; closed by mistake 0 of 1 |
| Research volume against a published hand-made evidence map (hyperemesis gravidarum, 2026-11-02) | Rank correlation 0.82 (0.42 to 0.99) over 10 questions; met its pre-set pass mark |
The weakest result is the first row: the tool often leaves an answered question open. Our errors sets out that failure and the others we have measured.
Cost per question
- Small run on one answered question: 1.69M tokens on average, over 4 questions. bench/SCORECARD.md:16
- Full run on one answered question: 5.45M tokens on average, over 2 questions (from the run log; approximate). bench/SCORECARD.md:73
- Full run on one known gap: 5.24M tokens (from the run log; approximate). bench/SCORECARD.md:109
How humans and the machine split the work
The machine sweeps, counts and checks. It reads more papers than a person could, never tires of logging a search, and refuses a claim it cannot quote. The person picks the questions, decides what matters, and makes the synthesis.
This is the project's working stance, not something we have shown. A system that recombines what it has read is good at finding what is missing between known pieces. It is less likely to make the leap that turns pieces into something new. Hand a machine all the music written before jazz, tell it to combine African rhythm with orchestral music, and it would probably not have produced jazz. That step came from people. So the tool hands its findings to a person, and the person decides what they add up to.
What we do not publish
Prompts and code are available to reviewers and researchers on request.