How PaperTrail works

The methodology behind every flag it raises — and why the numbers can be trusted.

What PaperTrail does

Researchers and regulated evidence teams cite dozens of primary sources under deadline pressure. Claims drift from what the source actually says: effect sizes get rounded up, caveats get dropped, and findings get stated more broadly than the trial established.

You submit a claim; PaperTrail traces it to the primary source in PubMed or ClinicalTrials.gov, extracts what the source actually found, and flags any discrepancy — with every flag anchored to the exact words in the source, and every number checked against the trial's own registered result.

The three-stage pipeline

A submitted claim moves through three agents in sequence; each stage caches its work.

  1. Stage 1

    Retrieval

    The claim is embedded with Voyage AI and matched against cached source records using pgvector similarity search. Sources fetched from PubMed or ClinicalTrials.gov are cached on first retrieval and never re-fetched. If no candidate clears the confidence threshold, retrieval abstains and returns no_support_found rather than forcing a match.

  2. Stage 2

    Extraction

    Claude reads the matched source and extracts a structured finding — effect size, population, condition, endpoint, caveats. Output is validated against a strict schema before use; raw model JSON is never trusted. Findings are cached per source, so a paper is read once.

  3. Stage 3

    Verification

    Claude compares the claim to the extracted finding and source, producing a discrepancy classification, trust score, explanation, and flagged spans. This is paired with a deterministic effect-size and registered-results cross-check that recomputes the trial's own numbers — a result that cannot wobble on resubmission and honestly defers when no numeric estimate exists.

The grounding guarantee

The verification model is asked to quote the source exactly, but nothing about a model response guarantees its quotes are real. PaperTrail makes that a code-enforced invariant: every flagged span is located inside the cached source before it is shown, and any span that cannot be located is dropped.

A span that cannot be pointed to in the source is, by definition, an unsourced claim about the source — so PaperTrail structurally cannot make one. This lives in lib/grounding.ts, covered by tests that fail loudly if the invariant is weakened.

Discrepancy taxonomy

Every verification resolves to exactly one of five outcomes.

Accurateaccurate
The claim faithfully represents the source's finding — right magnitude, population, and caveats. PaperTrail passes these cleanly, it does not flag everything.
Magnitude overstatedmagnitude_overstated
The claimed effect is larger than the source reports — a relative reduction quoted as absolute, a rounded-up percentage, or a benefit stated more strongly than the estimate supports.
Population overgeneralizedpopulation_overgeneralized
A finding shown in a specific population or subgroup is restated as if it applies more broadly than the trial established.
Caveat droppedcaveat_dropped
A material qualifier is missing from the claim — a safety signal, a confidence interval crossing null, or a limitation that changes how the result should be read.
No support foundno_support_found
Retrieval could not confidently match the claim to a primary source. PaperTrail returns this honest result rather than forcing a low-confidence match.

Honest limitations

The verification core is scoped to do one thing well. It deliberately does not:

  • One claim at a time, or a capped batch. It verifies a single claim per submission (or a bounded number of split sub-claims), not an entire 50-citation review in one pass.
  • Abstract and registered results as primary text. Extraction works from abstract/results text and the structured registry; the enterprise document pipeline adds full-PDF extraction on top.
  • Clinical-trial efficacy claims. It is not a general-purpose fact-checker for arbitrary or non-biomedical statements.
  • PubMed + ClinicalTrials.gov. The verification core searches these two — not preprint servers or press releases.

Explore

Verify a claim →Browse cached sources →See the evaluation results →Open the workspace →