Adding papers (ingestion)

An ingestion agent reads each paper and writes a structured record: the summary, links, workflow mapping, tools and marker definitions you see in the app. For now Claude does this by hand; an automated agent can take over later without changing the app.

Steps

Run these in the annotation_explorer folder.

See what's next. Lists the highest-ranked papers that haven't been ingested:
```
python pipeline/build_index.py --pending 10
```
Download the text. Saves the best free version (Europe PMC full text, the arXiv PDF, or the abstract) to sources/<id>/paper.md:
```
python pipeline/fetch_sources.py --next 5
```
or name papers: python pipeline/fetch_sources.py <id> <id>.
Agent reads and writes. The agent reads paper.md and writes ingested/<id>.json, following AGENT_INGEST.md.
Rebuild. Checks every record, re-ranks and regenerates the app's data:
```
python pipeline/build_index.py
```
Then reload the app.

A paper's id is its DOI with punctuation replaced by dashes, e.g. 10.1038/s41592-024-02235-4 → 10-1038-s41592-024-02235-4. A pending paper's abstract panel shows its id and these commands.

What a record contains

Part	Shown in the app as
`ingestion`	The source line: what was read, when, by which agent, and notes.
`enhanced_abstract`	One-line summary, sections, findings, limitations, why it matters.
`links`	Code, datasets and other links in the Links box.
`workflow`	The nine workflow steps and their involvement.
`algorithms`, `tools`	Lists with roles (proposed, used, compared).
`cell_types`	The marker definitions table.

The exact format is defined in schema/ingested_paper.schema.json. A record that doesn't match is reported by the build and left out of the app until it's fixed.

Rules the agent follows

Everything comes from the text it read, never from memory. What it couldn't read goes in the notes.
Links must appear in the paper or be checked to work.
Summaries are written in its own words.
Workflow steps it can't judge (e.g. abstract only) are marked unknown.
Baselines are listed with role compared.
Only explicit marker definitions go in cell_types; it leaves the list empty rather than guess.

The tool and method vocabulary

vocab/vocabulary.json lists standard names for tools and methods, with aliases, so usage can be counted across papers (e.g. SCANPY → Scanpy; Louvain (SNN graph) → SNN graph clustering). A trailing version (Seurat v4, Monocle 3) is split off automatically.

Names that aren't in the vocabulary are still counted under their own names, marked * in the app, and listed in cache/unlisted_names.json after each build. Add them to the vocabulary, as a new entry or as an alias of an existing one, when they're likely to come up again.

Where things are stored

Path	Committed?	Notes
`ingested/`	Yes	The agent's records.
`web/data/papers.json`	Yes	Built by `build_index.py`; don't edit by hand.
`cache/`	Yes	Citation histories and the unlisted-names report.
`sources/`	No	Downloaded full texts, which may not be redistributable.

Maintaining these help pages

The help is part of the app and should change with it:

Pages are plain HTML in web/help/, sharing help.css. The page list (and menu) is in help.js.
When a feature is added or changed, update the matching page and its "Last reviewed" date.
The app links to specific sections (help/<page>.html#<section>). Keep section ids stable, or update the links.
build_index.py runs pipeline/check_help.py, which reports broken help links and section anchors, links from the app to missing help sections, and pages missing from the menu. Run it on its own with python pipeline/check_help.py.
Avoid writing current counts into the help; the overview page reads them live from the data.