The complete guide to AI peer review in 2026

Peer review is the backbone of scientific publishing. It is also, by nearly every measure, too slow. The median time from submission to first decision at a mid-tier journal is measured in months, and at high-impact venues the wait regularly exceeds a year. For a researcher racing a grant deadline, or a clinician with a time-sensitive finding, that timeline is not a minor inconvenience, it is a structural barrier.

Science AI Journal addresses the latency problem without sacrificing rigor. Eight specialized AI agents, each calibrated on tens of thousands of real peer reviews, evaluate a manuscript in under 15 minutes and return a structured report: numeric scores, specific concerns tied to section numbers and figure labels, and a synthesis verdict. Human editors make the final acceptance decision.

This guide explains exactly how that works, and where it still falls short.

Why calibration is the whole story

The difference between an AI reviewer and a useful AI reviewer is calibration. A raw language model has read scientific text, but it has not internalized the tacit standards of a specific community: what NeurIPS considers an underpowered baseline, what eLife requires for a blinding statement, or what SciPost means when it flags "proof by example."

Our eight agents are calibrated against 69,000+ real peer reviews drawn from 19+ open peer-review platforms: OpenReview (ICLR, NeurIPS, ICML, TMLR, CVPR), eLife, BMJ Open, SciPost, PLOS ONE, Nature Communications, F1000Research, Royal Society Open Science, PubPeer, PREreview, Peer Community In, Qeios, Copernicus Publications, Open Research Europe, and ScienceOpen.

At inference time, each agent retrieves 8–40 of these real reviews, matched to the paper's discipline and study type, and prepends them to its prompt as calibration context. This retrieval-augmented approach means that when the methodology agent evaluates a clinical trial, it has just read how BMJ Open reviewers handle CONSORT violations; when it sees a deep-learning paper, it has read how NeurIPS reviewers flag missing ablations. The agent-type-to-query mapping is implemented in journal/src/lib/reviewers/training-rag.ts, which maintains specialized FTS5 search term sets for each of the eight agent roles.

A 2023 study on LLM-based feedback on research papers found that LLM feedback on NeurIPS submissions correlated meaningfully with human reviewer assessments, but only when the model had access to structured rubrics and calibration examples. Our system operationalizes exactly that finding: rubrics embedded in 658 lines of discipline-specific criteria (journal/src/lib/reviewers/prompts.ts) plus live retrieval of matched real reviews at every call.

Eight agents, one manuscript

Each agent specializes in a single dimension of review quality. They run sequentially, and each produces a numeric score from one to ten, a one-sentence verdict, and a detailed report. Every concern must cite a specific section number, figure, table, or equation from the manuscript. Generic statements ("the methodology is weak") are programmatically rejected.

Methodology. The most complex agent. It applies study-type-aware criteria: iso-compute baselines and seed averaging for ML papers; biological versus technical replicate distinctions for life-science work; CONSORT 2010 checklist items for clinical trials; STROBE criteria for observational studies; PRISMA requirements for systematic reviews; and theoretical guarantee standards for mathematics and physics. The full rubric covers six study types and scores concerns on a nine-point scale calibrated against eLife, NeurIPS, SciPost, and BMJ Open editorial standards.

Formulas & Equations. Checks mathematical derivations for internal consistency, notation clarity, and whether symbolic claims match the experiments that follow. Flags undefined notation and dimension mismatches.

Originality. Scores the paper's contribution relative to the literature it cites. Flags claimed novelty that is already documented in a cited reference, or a related-work section that understates prior art in the subfield.

Literature Coverage. Assesses whether the citation set is representative of the field. Identifies systematic gaps, for example citing only English-language work in a field with significant non-English output, or omitting a dominant methodological subfield entirely.

Reproducibility. Checks for code availability, random-seed documentation, full hyperparameter reporting, and dataset access statements. Uses the NeurIPS Reproducibility Checklist as a baseline for computational papers, and Royal Society Open Science data-accessibility requirements for empirical work.

Clarity & Language. Evaluates prose quality, sentence-level ambiguity, and structural coherence. Not a grammar checker: this agent scores whether the argument is followable and whether claims are precisely stated without hedging that makes them unfalsifiable.

Figures & Tables. Inspects axis labels, error bars on mean-value plots, self-contained table captions, and whether figures support the specific claims made in the results section. Flags figure panels that are referenced in the text but absent from the submission.

Prior Publication. Runs parallel checks across six bibliographic sources to detect previously published versions: CrossRef, Unpaywall, arXiv, medRxiv, bioRxiv, and our 4.4M-paper institutional library for local fuzzy matching. The confidence score blends title similarity (weighted at 60%) with abstract word-overlap (40%) when both are available, implemented in journal/src/lib/reviewers/prior-publication.ts. See our prior publication detection deep dive for the full algorithm.

After all eight agents complete, a synthesis agent reads every individual report and produces a final overall score, an editorial recommendation, and a ranked list of the concerns that most need addressing before resubmission.

Pre-submission scoring: find the major issues first

Before sending a manuscript for full review, authors can run the abstract and title through our pre-submission scorer. The scorer returns a Tier 1–5 estimate and a plain-language summary of the largest risks the full review is likely to surface.

The five tiers correspond to real editorial patterns from the platforms that trained the system:

• Tier 1: Clear primary outcomes, pre-registration or justified absence of it, complete CONSORT or STROBE reporting, accessible data, robust controls.
• Tier 2: Minor gaps (missing power analysis, one absent control) that do not invalidate the central result.
• Tier 3: Significant issues requiring major revision before the paper is defensible.
• Tier 4: Fundamental methodological flaws; rejection likely without redesigning a study component.
• Tier 5: Results uninterpretable as written: data leakage, HARKing, undisclosed fabrication indicators, or causation claimed from correlational data without a causal inference framework.

A Tier 3 result is not a rejection notice. It is a ranked list of what to fix before submitting anywhere, not only to us.

What AI review cannot do

Honesty about limits is part of our methodology.

Novelty judgment at the frontier. Calibration against historical reviews tells an agent what was novel two or three years ago. Whether a result advances the genuine frontier of a rapidly moving field requires contextual judgment, awareness of unpublished concurrent work, and community consensus that no retrieval system can fully replicate. We flag potential contribution gaps; we do not certify novelty.

Social and ethical framing. Is the research question itself worth pursuing? Does the study design encode a harmful assumption? These questions require domain expertise, lived experience, and community discourse. They are outside the scope of any automated system operating on text alone.

Detecting image manipulation or data fabrication. Our Figures & Tables agent checks for axis-label issues, missing error bars, and figure-caption mismatches. It does not perform forensic detection of pixel-level image manipulation or statistical anomalies that signal data fabrication. Those cases require dedicated tools and human forensic expertise.

Distinguishing exploratory from post-hoc analysis. HARKing (Hypothesizing After Results are Known) leaves structural signatures in a manuscript that our methodology agent flags. But distinguishing a genuinely exploratory finding from a post-hoc claim presented as a priori requires knowledge of the authors' workflow that is not available in the submitted text.

We document these limits because researchers deserve accurate descriptions of the tools they use. If any of these cases apply to your submission, we recommend pairing AI review with a domain expert's judgment before drawing firm conclusions from the report.

What this means for authors

The practical workflow: run the pre-submission scorer first to surface structural issues before full review consumes credits. If the scorer flags a Tier 4 concern, address it and re-run. When the manuscript is in good shape, submit for full review; all eight agents return a structured report within 15 minutes.

For researchers trying to identify which question to pursue next, our research-gap finder indexes 17,000+ extracted gaps synthesized from 250M+ papers via OpenAlex. It is a starting point for the kind of targeted literature scan that otherwise takes weeks to do manually.

All review reports produced by Science AI Journal are released under CC BY 4.0.

Frequently asked questions

What disciplines does Science AI Journal cover? Ten disciplines: Engineering, Life Sciences, Physical Sciences, Computer Science, Medicine, Mathematics, Environmental Science, Social Science, Humanities, and Economics. The methodology agent automatically selects the appropriate study-type criteria based on the paper's declared field and abstract content.

How are the 69,000+ training reviews used at inference time? Each agent queries the training database using FTS5 full-text search against a curated term list specific to that agent's role, then prepends the 8–40 most relevant real reviews to the prompt as calibration context. The retrieval is discipline-aware: a clinical trial triggers CONSORT-specific search terms; an ML paper triggers OpenReview-specific terms for seed averaging and ablation coverage.

Can a paper score well on the pre-submission scorer and still get a poor full review? Yes. The scorer reads only the abstract and applies fast local heuristics. The full review agents read the entire manuscript and apply detailed rubrics. A paper can have a well-structured abstract and a deeply flawed methods section. The scorer is a triage tool, not a substitute for full review.

Does the AI make the final publication decision? No. Human editors make acceptance decisions. The eight agents produce structured evidence, scores, and ranked concerns; the editorial board weighs that evidence alongside any disciplinary context the agents cannot access.

How does the Prior Publication agent handle preprints? Preprints on arXiv, medRxiv, or bioRxiv are not automatically treated as prior publication. The agent flags them as related prior versions and reports the confidence score so editors can judge whether the submission represents substantive new work or a verbatim repost.

Reviews are performed by AI agents. Editorial decisions are made by human editors.