How AI Review's Agents Are Calibrated with Open Peer Review

An AI reviewer that has never read open peer review is guessing at what editors and referees care about. AI Review does not leave its reviewers to guess: before each of seven language-model agents writes a word of its report, it is shown up to 60 real examples from open peer review. They are drawn from an index of 5,000+ peer-review texts collected from 15 open-review platforms, though, as the selection works today, from only four of those platforms. This post explains where the texts come from, how the examples reach each agent at review time, and what that kind of calibration can and cannot do.

Why calibration is the right word

Most discussion of AI reviewers focuses on the underlying language model. That misses much of the problem. A capable model that has never seen open peer review has no instinct for the biological-replicate rule (n = 3 technical replicates is not n = 3 biological replicates), for what machine-learning reviewers mean by "seed averaging", or for the point where a SciPost referee says "you conjectured, not proved". Those are domain norms, and they live in actual review records.

Calibration, as we use the term, is not fine-tuning. The model's weights are never changed. Instead, at the moment an agent runs, its prompt is prefixed with real examples from open peer review chosen for that agent's job. The examples are there to shape what the agent notices, what it calls a major concern, and how it anchors its 1–10 score; we have not measured how much they do. This is a form of retrieval-augmented generation (RAG) as described by Lewis et al. in "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks" (arXiv:2005.11401, 2020), with one difference: the retrieved documents are not background facts but examples of reviewers' judgement.

Where the corpus comes from

The corpus is built from review records that the platforms themselves publish openly. The table counts them in distinct texts — each referee report, reviewer's follow-up, editor's decision or assessment, public comment or authors' response once, however many times our index stores it — as measured on 28 October 2026 by reading a seeded sample of each platform's texts by hand. Where a platform mixes review text with other material (OpenReview's procedural messages, the bot and editor chatter in JOSS review threads), the figure is the sampled share of its texts; elsewhere every text was read or classified.

The total is an estimate from those samples; its conservative floor is 5,092, which is why we quote 5,000+. Earlier versions of this post counted stored rows instead, but the index keeps a typical review eight or nine times (once per agent's slice of the corpus, plus re-imported copies), so the row count overstated the corpus about thirteen-fold.

Several kinds of record in the same search index are not in this table or in the 5,000 figure: the text we stored from PLOS ONE, BMJ Open, ScienceOpen and the Royal Society, which turned out to be article text, metadata or rating-only stubs rather than reviews; text extracted from papers in our library; sample reviews written as examples; and three reviewer reports on a paper of our own, added by hand. None of it is counted as reviews. Since 28 October 2026 the agents' examples are drawn only from the 15 counted platforms, so the article text, the library text and our own reports cannot reach an agent; the sample reviews are stored under counted platforms and are not filtered out, though none was drawn in the replay described below.

Each record is stored with its source platform, its review type and section (a full report, or its weaknesses, strengths, questions or summary), a quality tier, and an outcome label: accept, revise or reject. Where a platform publishes the editorial decision, the label is that decision; on several platforms — PREreview, PeerJ, Copernicus and the PCI communities among them — our scrapers infer the label from the review's own wording or set a default, and PREreview has no editorial decision at all. Counted in distinct texts on 28 October 2026, 86% carry a label; the rest are marked unknown. The labels lean towards acceptance — about 58% of the texts are labelled accept — which matters for how examples are chosen.

How an agent's examples are chosen

Eight of AI Review's nine agents are language-model agents. Seven of them read examples from this corpus: Methodology, Formulas & Equations, Originality, Literature Coverage, Reproducibility, Clarity & Language, and Figures & Tables. The eighth, Research Gap (added on 25 October 2026), reads comparable published papers retrieved for the manuscript's topic instead. They run in parallel. The ninth, Prior Publication, is a deterministic lookup and uses no examples at all (more on it below).

For each of those seven agents, the server searches the corpus with a full-text index (SQLite FTS5, ranked by BM25). The search is built from that agent's own review vocabulary, between about 100 and 185 terms per agent: the methodology agent searches for terms such as "power analysis", "allocation concealment", "pseudoreplication" and "HARKing"; the figures agent for error bars, axis labels and colour-blind-safe palettes. The examples are chosen per agent, not per manuscript: your paper is what the agent reviews, and the examples show it the standard to review against.

From the candidates the search returns, a selection step then:

• keeps only texts from the 15 counted platforms, one copy of each text, and only reviewers' form sections (weaknesses, strengths, summary or questions) — or whole reports from the platforms where at least nine in ten of the texts we read by hand were reviewer reports;
• drops texts that are authors' rebuttals, and page leftovers, that the scrapers filed as review sections, and excerpts too short to teach anything (an example can still carry a short excerpt of the authors' response to it, appended after the review and marked as the authors' response);
• prefers the "weaknesses" sections of high-quality texts, since all seven agents look for flaws, and restricts the Methodology, Formulas, Originality and Figures agents to texts rated high, curated or standard quality;
• aims for roughly 40% reject, 30% revise, 20% accept and 10% unknown labels, so an agent sees what gets papers rejected, not only what gets them accepted;
• lets no platform supply more than 40% of an agent's examples unless the other platforms run out, so OpenReview, the largest source, does not crowd out the rest.

When the agents run on Claude, as they do in production, each agent call carries up to 60 examples, each trimmed to 3,000 characters. Because the selection is the same for every manuscript, we can say exactly what it produces. In a read-only replay on 28 October 2026, each agent's 60 examples were distinct texts, all filed as weaknesses sections, and they came only from four of the 15 platforms: OpenReview, SciPost, PeerJ and JOSS. None came from eLife, Nature Communications, PREreview, Copernicus, Open Research Europe or the PCI communities, whose texts did not rank among any agent's candidates. The 40% limit held wherever other platforms still had candidates; where they did not, it gave way: only OpenReview and SciPost texts ranked among the Formulas agent's candidates, so SciPost supplied 36 of its 60, and OpenReview supplied 28 of the Originality agent's 60. In an eye-read sample of the examples, about one in eight was still a reviewer's follow-up, an editor's message or an authors' reply that the filters missed, which is why we call them examples from open peer review rather than peer reviews.

What each agent is asked to check

The examples calibrate; the agent's own prompt sets the checklist. Each language-model agent returns a score from 1 to 10 with strengths, concerns, questions for the authors and a written justification that ties the score to a rubric.

Methodology carries venue-specific checklists — machine-learning venues (baselines, data leakage, seeds, ablations), eLife and the life sciences (biological replicates, blinding, randomisation), BMJ Open and clinical research (CONSORT, STROBE, PRISMA, GRADE), SciPost for physics and mathematics, Copernicus for earth and climate science, the PCI communities for ecology and evolution, and more — plus general red flags such as causation claimed from correlation, pseudoreplication and HARKing. Its rubric puts rigorous, well-controlled, reproducible designs at 9–10 and fundamental flaws such as data leakage or HARKing at 3–4.

Formulas & Equations checks derivations, dimensional consistency, conflicting notation and undefined variables, and proposes corrections.

Originality checks the manuscript's novelty claims against papers in our 7.9M-paper local library, and looks for text-level integrity signals, including statistics that are arithmetically impossible, such as GRIM-test failures.

Literature Coverage evaluates citation completeness, missing seminal references and self-citation balance (it flags self-citation above 30%), with a live OpenAlex snapshot of the field for context.

Reproducibility asks whether code and data are available and versioned, whether random seeds and compute are reported, and whether the methods are detailed enough for an independent lab to repeat the work.

Clarity & Language checks structure, abstract completeness (background, objective, methods, key results with numbers, and implications), undefined acronyms, overclaiming, and conventions such as qualifying every use of "significant".

Figures & Tables checks captions, defined error bars, readable axes, colour-blind-safe palettes, and whether small samples are shown as individual data points.

Prior Publication is not a language model. It fans out in parallel to seven external sources — CrossRef, PubMed, arXiv, bioRxiv, medRxiv, Europe PMC and Unpaywall — and to our local library, with a 12-second timeout per source, looking for title and abstract overlap. Our prior publication detection post explains how.

A final synthesis step reads the agents' reports and returns a verdict — Accept, Minor Revision, Major Revision or Reject — with the issues that must be addressed, alongside an overall score built from the agents' 1–10 scores.

What calibration cannot fix

Calibration on past review texts does not confer knowledge the corpus does not contain.

The Originality agent compares against our local library, not the whole literature, so it cannot see verbatim copying from a paper the library does not hold. The Prior Publication check is limited to what its sources and the library have indexed.

Coverage is uneven. OpenReview supplies about 54% of the peer-review texts in the index, reflecting the culture of open review in machine learning, and the examples the agents are shown come from fewer platforms still. Clinical research is thin. The text we stored from BMJ Open turned out to be article text, so the clinical and health review text in the corpus comes mainly from PeerJ, where about a quarter of the texts concern clinical or health studies (by a keyword check of the papers' titles and abstracts), and from PREreview's reviews of health preprints; Open Research Europe is multidisciplinary, not a clinical platform. Because the examples are not matched to the manuscript, a clinical trial is reviewed against the same examples as a machine-learning paper, and small fields may have review norms that no platform in the corpus captures.

And calibration cannot fix a manuscript. An agent can tell you that Section 3.2 has no power analysis; it cannot run one for you. The report is a diagnosis, not a rewrite.

What this means for authors

If you are deciding whether to trust an AI review, the useful question is not "is it an AI?" but "what was it calibrated against, and how?" The sources above, the counts, and the way examples reach each agent are all documented here and on our dataset page.

When a full manuscript is ready, AI Review runs all nine agents over the PDF; it is paid in credits (see pricing). For a quicker read before that, Pre-Check is a separate, credit-metered tool: from a title and abstract it estimates acceptance odds for top, mid and open/emerging journals, based on where the most similar papers in our local library were published — it does not use this review corpus. For exploratory work, the Research Gap Finder is free to search, and unlocking a topic writes its research gaps from our 7.9M-paper local library.

Frequently asked questions

Is the model trained on these texts? No. The model's weights are not changed. Real examples from open peer review are placed in an agent's prompt each time it runs, which is why this is called calibration by retrieval rather than training.

Does an agent see different examples for different manuscripts? Not today. Each agent's search is built from its own review vocabulary, not from your manuscript, so the methodology agent is calibrated on methodology critiques whatever the paper's field. The examples change as the corpus grows.

How does the reject/accept balance affect the scores? The labels in the corpus lean towards accept, so the selection deliberately over-samples reject labels. An agent calibrated only on texts labelled accept would miss the patterns that get papers rejected; one calibrated only on rejections would over-flag solid work. We have not measured how much the balance moves the scores.

Can I see which examples influenced a review of my paper? Not in the report today. The report shows each agent's own findings, score and justification.

Where do the examples come from, and who owns them? From peer-review texts that the platforms above publish openly; in the 28 October 2026 replay every example the seven calibrated agents were shown came from OpenReview, SciPost, PeerJ or JOSS. Every record keeps its source platform, and we do not present the texts as our own writing. Licence terms differ from platform to platform; our dataset page describes what we share and on what terms.

AI Review reports are written by AI agents. They are advice to the author, not an editorial decision, and nothing is sent to a journal on your behalf.