The complete guide to AI peer review in 2026

Updated 26 October 2026. This post was written when AI Review ran eight agents. Since 25 October 2026 it runs nine: a Research Gap agent now checks the gap a manuscript claims against comparable published papers. Seven of the eight language-model agents are calibrated on the peer-review corpus; the Research Gap agent reads published papers instead, and Prior Publication is a lookup.

Peer review is the backbone of scientific publishing. It is also, by nearly every measure, too slow. The median time from submission to first decision at a mid-tier journal is measured in months, and at high-impact venues the wait regularly exceeds a year. For a researcher racing a grant deadline, or a clinician with a time-sensitive finding, that timeline is not a minor inconvenience, it is a structural barrier.

Science AI Journal addresses the latency problem without sacrificing rigor. Nine specialized AI agents, seven of them calibrated against an index of 5,000+ real peer-review texts, evaluate a manuscript in under 15 minutes and return a structured report: numeric scores, specific concerns tied to section numbers and figure labels, and a synthesis verdict. It is pre-submission feedback: the acceptance decision stays with the editors of the journal you submit to.

This guide explains exactly how that works, and where it still falls short.

Why calibration is the whole story

The difference between an AI reviewer and a useful AI reviewer is calibration. A raw language model has read scientific text, but it has not internalized the tacit standards of a specific community: what NeurIPS considers an underpowered baseline, what eLife requires for a blinding statement, or what SciPost means when it flags "proof by example."

Seven of our language-model agents are calibrated against an index of 5,000+ real peer-review texts collected from 15 open-review platforms: OpenReview (ICLR, NeurIPS, ICML, TMLR, CVPR), SciPost, PREreview, PeerJ, Copernicus Publications, the Journal of Open Source Software, Open Research Europe, Peer Community In, Nature Communications, eLife, Review Commons, PubPeer, F1000Research, Qeios and EMBO Press. Each referee report, editor's decision, public comment or authors' response counts once, however many times our index stores it.

At inference time, each of those seven agents retrieves up to 60 real examples from open peer review, chosen for that agent's specialty — the methodology agent reads methodology critiques, the figures agent figure critiques — weighted toward the highest-quality texts, mixed across outcome labels, with no platform above 40% of an agent's examples unless the others run out, and prepends them to its prompt as calibration context. The examples are matched to the agent's job, not to your paper's topic. On 28 October 2026 every agent's examples came only from four of the platforms: OpenReview, SciPost, PeerJ and JOSS. The agent-type-to-query mapping is implemented in journal/src/lib/reviewers/training-rag.ts, which keeps a specialized FTS5 search-term set for each of the seven calibrated agents, plus a general set for the synthesis pass.

A 2023 study on LLM-based feedback on research papers found substantial overlap between GPT-4's feedback on papers and the points human reviewers raised. Our system adds two things a general-purpose model lacks: discipline-specific rubrics written into each agent's prompt (journal/src/components/hakem/ScienceAI.tsx) and live retrieval of real examples from open peer review at every call.

Nine agents, one manuscript

Each agent specializes in a single dimension of review quality. They run in parallel, and each produces a numeric score from one to ten, a one-sentence verdict, and a detailed report. Every concern must quote or point to a specific passage, section, figure, table, or equation in the manuscript; generic statements ("the methodology is weak") are ruled out in the instructions of every language-model agent.

Methodology. The most complex agent. It applies study-type-aware criteria: iso-compute baselines and seed averaging for ML papers; biological versus technical replicate distinctions for life-science work; CONSORT 2010 checklist items for clinical trials; STROBE criteria for observational studies; PRISMA requirements for systematic reviews; and theoretical guarantee standards for mathematics and physics. The rubric is keyed to the study design the agent detects, and its one-to-ten scale follows the editorial standards of eLife, NeurIPS, SciPost and BMJ Open.

Formulas & Equations. Checks mathematical derivations for internal consistency, notation clarity, and whether symbolic claims match the experiments that follow. Flags undefined notation and dimension mismatches.

Originality. Scores the paper's contribution relative to the literature it cites. Flags claimed novelty that is already documented in a cited reference, or a related-work section that understates prior art in the subfield.

Literature Coverage. Assesses whether the citation set is representative of the field. Identifies systematic gaps, for example citing only English-language work in a field with significant non-English output, or omitting a dominant methodological subfield entirely.

Reproducibility. Checks for code availability, random-seed documentation, full hyperparameter reporting, and dataset access statements. Uses the NeurIPS Reproducibility Checklist as a baseline for computational papers, and Royal Society Open Science data-accessibility requirements for empirical work.

Clarity & Language. Evaluates prose quality, sentence-level ambiguity, and structural coherence. Not a grammar checker: this agent scores whether the argument is followable and whether claims are precisely stated without hedging that makes them unfalsifiable.

Figures & Tables. Inspects axis labels, error bars on mean-value plots, self-contained table captions, and whether figures support the specific claims made in the results section. Flags figure panels that are referenced in the text but absent from the submission.

Research Gap. Quotes the gap the manuscript says it fills, checks it against comparable published papers retrieved from our library (with their mined methods, data and results where we hold them), and scores whether that gap is real and addressed. It reads those papers instead of calibration examples.

Prior Publication. Runs parallel checks to detect previously published versions: CrossRef, PubMed, Europe PMC, Unpaywall, arXiv, medRxiv, bioRxiv, and our 7.9M-paper local library for local fuzzy matching. It is a database lookup, not a language model. The confidence score blends title similarity (weighted at 60%) with abstract word-overlap (40%) when both are available, implemented in journal/src/lib/reviewers/prior-publication.ts. See our prior publication detection deep dive for the full algorithm.

After all nine agents complete, a synthesis agent reads every individual report and produces a final overall score, a recommendation, and a ranked list of the concerns that most need addressing before you submit.

Pre-Check: a fast read on fit, before the full review

Before running the full review, authors can put the title and abstract through Pre-Check. It reads only those two fields and answers in seconds: acceptance-probability estimates for three journal tiers (top, mid, and open or emerging), based on where papers like yours were published, the field it detects, a ranked shortlist of target journals, and a prior-publication scan.

What it cannot do is judge the manuscript itself. Whether the methods report allocation concealment, whether the figures carry error bars, whether a claim outruns its evidence: those questions need the full text, which is what the language-model agents read. Treat a Pre-Check tier as a question about fit, and the full review as the ranked list of what to fix before you submit anywhere.

What AI review cannot do

Honesty about limits is part of our methodology.

Novelty judgment at the frontier. Calibration against historical reviews tells an agent what was novel two or three years ago. Whether a result advances the genuine frontier of a rapidly moving field requires contextual judgment, awareness of unpublished concurrent work, and community consensus that no retrieval system can fully replicate. We flag potential contribution gaps; we do not certify novelty.

Social and ethical framing. Is the research question itself worth pursuing? Does the study design encode a harmful assumption? These questions require domain expertise, lived experience, and community discourse. They are outside the scope of any automated system operating on text alone.

Detecting image manipulation or data fabrication. Our Figures & Tables agent checks for axis-label issues, missing error bars, and figure-caption mismatches. It does not perform forensic detection of pixel-level image manipulation or statistical anomalies that signal data fabrication. Those cases require dedicated tools and human forensic expertise.

Distinguishing exploratory from post-hoc analysis. HARKing (Hypothesizing After Results are Known) leaves structural signatures in a manuscript that our methodology agent flags. But distinguishing a genuinely exploratory finding from a post-hoc claim presented as a priori requires knowledge of the authors' workflow that is not available in the submitted text.

We document these limits because researchers deserve accurate descriptions of the tools they use. If any of these cases apply to your submission, we recommend pairing AI review with a domain expert's judgment before drawing firm conclusions from the report.

What this means for authors

The practical workflow: run Pre-Check first to check fit and prior publication from the title and abstract. Then run the full AI Review on the manuscript; all nine agents return a structured report within 15 minutes. Address the concerns it ranks highest before you submit.

For researchers trying to identify which question to pursue next, our research-gap finder holds 1,100,000+ open gaps mined from 780,000+ papers in a 7.9-million-paper local library, and searching a topic there is free. It is a starting point for the kind of targeted literature scan that otherwise takes weeks to do manually.

Frequently asked questions

What disciplines does Science AI Journal cover? Ten disciplines: Engineering, Life Sciences, Physical Sciences, Computer Science, Medicine, Mathematics, Environmental Science, Social Science, Humanities, and Economics. The methodology agent automatically selects the appropriate study-type criteria based on the paper's declared field and abstract content.

How are the calibration texts used at inference time? Each of the seven calibrated agents queries the calibration index using FTS5 full-text search against a curated term list specific to that agent's role, then prepends up to 60 of the best-matching real examples from open peer review to the prompt as calibration context. The search terms are fixed per agent role, not taken from your paper: the methodology agent's terms cover reporting standards, statistics and study design across fields.

Can a paper score well on Pre-Check and still get a poor full review? Yes. Pre-Check reads only the title and abstract and applies fast local retrieval. The full review agents read the entire manuscript and apply detailed rubrics. A paper can have a well-structured abstract and a deeply flawed methods section. Pre-Check is a triage tool, not a substitute for full review.

Does the AI decide whether my paper is accepted? No. Acceptance is decided by the editors of the journal you submit to. The agents produce structured evidence, scores, and ranked concerns for you to act on before you submit. Our own journal publication route is planned; it is not open for submissions yet.

How does the Prior Publication agent handle preprints? Preprints on arXiv, medRxiv, or bioRxiv are not automatically treated as prior publication. The agent flags them as related prior versions and reports the confidence score, so you can judge, before a journal's editors do, whether the submission reads as substantive new work or a verbatim repost.

AI Review reports are written by AI agents. They are pre-submission feedback, not an editorial decision.