Updated 4 min readengineering

Peer review in 15 minutes: how Science AI Journal works

An inside look at our multi-agent review engine (nine agents since October 2026): what each agent checks, why narrow reviewers beat one broad prompt, and what the report does not claim.

By Science AI Journal Editorial

Updated 26 October 2026. This post was written when AI Review ran eight agents. Since 25 October 2026 it runs nine: a Research Gap agent now checks the gap a manuscript claims against comparable published papers. Seven of the eight language-model agents are calibrated on the peer-review corpus; the Research Gap agent reads published papers instead, and Prior Publication is a lookup.

Traditional peer review takes between 2 and 18 months, depending on field. For a PhD student with a tight submission deadline — or a clinician with a finding that might change practice — that timeline is incompatible with how science actually moves today.

AI Review runs a manuscript through nine specialised AI review agents; seven of them are calibrated with up to 60 real examples from open peer review each, drawn from an index of 5,000+ real peer-review texts collected from 15 open-review platforms; on 28 October 2026 the seven calibrated agents' examples came only from four of them: OpenReview, SciPost, PeerJ and JOSS. A manuscript uploaded in the morning comes back well before lunch with a report from every agent and a synthesis recommendation — feedback to act on before you submit, not an editorial decision.

What each agent checks

AgentFocusHow it works
MethodologyStudy design, sample-size justification, CONSORT / STROBE / PRISMA compliance, seeds and baselines for ML work, causal validityLanguage-model agent; the rubric is keyed to the study design it detects
Formulas & EquationsDerivations, notation, dimensional consistencyLanguage-model agent; every concern must quote the equation
OriginalityNovelty claims against the prior work the paper cites, self-plagiarism signalsLanguage-model agent
Literature CoverageWhether the references present are representative, current, and support the claims they are attached toLanguage-model agent; a deterministic check then flags any author-year, DOI or cite-key it names that is not in the manuscript
ReproducibilityCode and data availability, seeds, hyperparameters, protocol detailLanguage-model agent
Clarity & LanguageWhether the argument is followable and the claims are precisely statedLanguage-model agent
Figures & TablesAxis labels, error bars, caption sufficiency, figure–claim matchLanguage-model agent; reads the figure images when they are supplied
Research GapWhether the gap the manuscript says it fills is real and addressedLanguage-model agent; reads comparable published papers retrieved from our library instead of calibration examples
Prior PublicationTitle and abstract matched against CrossRef, PubMed, Europe PMC, Unpaywall, arXiv, medRxiv, bioRxiv and our local 7.9M-paper FTS5 index, in parallelDatabase lookup — no language model
SynthesisWeighs the agents' reports and writes the overall recommendation and scoreLanguage-model pass over the agent reports

Why nine and not one?

A single monolithic prompt hits two walls. First, it hallucinates — it wants to find issues everywhere, so it invents them. Second, it can't hold all the relevant context: methodology rubrics for every study design, plus statistics guidance, plus figure-reading heuristics, plus calibration examples for each, exceed any single prompt's useful attention.

The agent pattern lets each reviewer carry only the rubric it needs and only the calibration examples matching its specialty. We have not published a measured agreement rate between the agents and human editorial decisions, and we will not quote a number we have not run; what we have measured is on the benchmarks page.

What you get back

Every agent's full report, not just a score: what each one flagged, the manuscript passage it points to, and the synthesis that weighs them. It is yours to act on before you submit to any journal. A journal publication route, with the review report published beside each paper, is planned; it is not open for submissions yet, and we have not published any papers. Open access without open review is transparency theatre; when that route opens, we think the two have to ship together.

What we won't claim

  • We do not replace human peer review for stakes where it genuinely matters — drug trials, regulatory submissions, grant panels.
  • We do not outperform a careful, well-resourced human reviewer on nuanced theoretical work.
  • We do not generate novel scientific insight. We review.

What we do claim: for most manuscripts, a competent, fast, transparent first pass before a journal's referees see them is a better default than waiting months to learn about problems you could have fixed.

→ Run an AI Review · Run a Pre-Check

#ai-peer-review#open-access

Related posts

Command palette

Jump anywhere, run any action.