Methodology
How Science AI Journal performs rigorous peer review in under 15 minutes — the review corpus, agent calibration, and the limits of what we claim.
The 5-step pipeline
- 1
Submission & intake
You upload a manuscript as PDF, Word (.doc/.docx) or LaTeX (.tex). The intake step extracts the title, abstract, body text and references (plus any images embedded in a Word file), and flags a low-quality PDF extraction so you can re-upload a cleaner file.
- 2
Prior-publication fan-out
Before any agent runs, the title and abstract fan out in parallel to 7 external sources — CrossRef, PubMed, arXiv, bioRxiv, medRxiv, Europe PMC and Unpaywall — and to our 7.9M-paper local library, with a 12-second timeout per source. Matches are surfaced with a confidence level and a link, so you can judge them yourself.
- 3
Calibration retrieval
For each of the 7 calibrated language-model agents (all but Research Gap, which reads comparable published papers instead), up to 60 real examples from open peer review are retrieved from a SQLite FTS5 index that holds 5,000+ real peer-review texts collected from 15 open-review platforms (OpenReview, SciPost, PREreview, PeerJ, Copernicus and eLife, among others). The retrieval is concern-aware, not manuscript-aware: each agent pulls examples that match its own rubric, the same for every paper, weighted towards higher-quality texts and a mix of accept, revise and reject labels, with no platform above 40% of an agent's examples unless the others run out. On 28 October 2026 every agent's examples came only from OpenReview, SciPost, PeerJ and JOSS.
- 4
Parallel agent review
The 9 agents — methodology, formulas, originality, literature coverage, reproducibility, clarity, figures, research gap, and prior publication — start at the same time. Each produces a score, a qualitative summary, and a structured report. The 8 language-model agents run on Claude Sonnet; the Prior Publication agent is a deterministic lookup, not a language model.
- 5
Synthesis & editorial decision
A synthesis pass reconciles the 9 agent reports into a single recommendation (accept, minor revision, major revision, reject) with a numeric score. Every agent's full report stays visible next to it. A journal publication route — with the review report published alongside the paper — is planned; it is not open yet.
Review corpus provenance
7 of the 8 language-model agents are calibrated with real examples from open peer review, drawn from texts collected from public open-review pages. Counted once each — a referee report, a reviewer’s follow-up, an editor’s decision or assessment, a public comment or the authors’ response — the peer-review texts total 5,000+ from 15 open-review platforms, largest first:
- OpenReview (ICLR, NeurIPS, ICML and more)
- SciPost
- PREreview
- PeerJ
- Copernicus (open discussion)
- Journal of Open Source Software
- Open Research Europe
- Peer Community In
- Nature Communications
- eLife
- Review Commons
- PubPeer
- F1000Research
- Qeios
- EMBO Press
The figure counts collected texts only: no synthetic data and no proprietary publisher corpora. It was measured on 2026-10-28 by reading a seeded sample of each platform’s texts by hand; the per-platform table is on the dataset page. The index the agents retrieve from stores each text several times, once per agent’s slice, and besides these texts it holds other text stored with them (article and front-matter text from journal pages and from papers in our library, which we do not count as reviews), sample reviews written as examples rather than collected, and three reviewer reports on a paper of our own; none of that is counted, and the agents’ examples are drawn only from the 15 counted platforms.
Aggregated per-agent example files back the FTS5 retrieval index used during review.
Agent specialisations
9 specialist agents. 7 of the language-model agents are each shown up to 60 real examples from open peer review, found by searching the calibration corpus with that agent’s own review vocabulary; the Research Gap agent reads comparable published papers instead, and the Prior Publication agent is a deterministic lookup.
Methodology
Audits study design, statistical power, and analytical choices against field-specific rigour standards (CONSORT, STROBE, PRISMA).
Formulas & Equations
Verifies mathematical derivations, checks dimensional analysis, and flags algebraic errors.
Originality
Checks the manuscript's novelty claims against papers in our 7.9M-paper local library and flags possible overlap and self-plagiarism signals.
Literature Coverage
Evaluates citation completeness, missing seminal references and self-citation balance, with a live OpenAlex snapshot of the field (volume, top venues, peak year) for context.
Reproducibility
Inspects code availability, dataset accessibility, and sufficiency of methods detail for independent replication.
Clarity & Language
Assesses readability, structural flow, and adherence to scholarly writing norms.
Figures & Tables
Checks figure quality, caption completeness, and appropriateness of visual encodings.
Research Gap
Quotes the gap the manuscript says it fills, checks it against comparable published papers retrieved from our library (with their mined methods, data and results where we hold them), and scores whether that gap is real and addressed.
Prior Publication
A deterministic lookup, not a language model: fans out in parallel to 7 external sources — CrossRef, PubMed, arXiv, bioRxiv, medRxiv, Europe PMC and Unpaywall — and our local library to detect prior publication and duplicate submission.
What we don't claim
AI peer review is not a replacement for domain-expert human review in high-stakes settings (clinical trials, safety-critical systems, paradigm-shift claims). We're transparent about the limits:
- Agents can miss subtle methodological flaws that require cutting-edge domain knowledge.
- Prior-publication detection is excellent for exact overlap but weaker for paraphrased or translated duplicates.
- Originality scoring depends on index coverage — niche non-English work may be under-indexed.
- The synthesis recommendation is a starting point, not a verdict: the decision stays with you and the journal you submit to.