How AI Peer Review Scores Map to Real Acceptance Rates

Science AI Journal scores every paper on a 1-10 scale across eight independent agents. That scale is calibrated against 69,000 real peer reviews from 19 platforms, including OpenReview (NeurIPS), eLife, BMJ Open and SciPost, where reviewers' words and editors' decisions are documented publicly. This post traces the calibration chain from those real decisions to the number in your review report.

Why calibration is the hard part

A score without calibration is just a preference. If an AI model has never encountered a real NeurIPS review, its "7 out of 10" reflects a vague impression of quality, not a reproducible signal grounded in how journals actually evaluate papers.

Calibration connects our 10-point scale to the documented distribution of outcomes at real journals. When the methodology agent assigns a 5, it draws on examples from the 69,000-review corpus where reviewers writing about the same class of problem were associated with major-revision recommendations. When it assigns a 3, the examples it draws on reflect near-reject language from the same platforms. The score is not an opinion; it is an interpolation into a well-populated reference distribution.

The 10-point scale and its empirical anchors

Each of the eight agents uses the same banded rubric, derived from the stated standards of the platforms we calibrate against. The methodology agent's rubric documents the bands explicitly:

The same structure appears in every agent's rubric. A well-written paper can score 8 on Clarity and 3 on Methodology. Those scores are independent signals; they do not average into a comfortable 5.5.

How 69,000 real reviews anchor each band

The calibration corpus covers 19 platforms that publish their peer reviews openly, including:

• OpenReview (NeurIPS, ICLR, ICML, AISTATS, TMLR, EMNLP, ACL, COLM, MIDL, CoRL, CVPR)
• eLife, BMJ Open, SciPost, PLOS ONE, Nature Communications
• Copernicus Publications, F1000Research, PREreview, Qeios, PubPeer
• Peer Community In (Ecology, Evolutionary Biology, Mathematical & Computational Biology, Neuroscience)

At review time, each agent receives real examples from this corpus, retrieved by content similarity to the paper being reviewed. The examples show what a "seed averaging" concern looks like in an actual NeurIPS review, what "biological replicates" means to a real eLife reviewer, and what a SciPost referee means when they write "you conjectured, not proved." These are not paraphrases; they are the documented language of the relevant community.

This is retrieval-augmented generation, not fine-tuning. The model is not retrained on these reviews; they arrive as context at inference time. This keeps calibration current as the corpus grows, without a separate training cycle.

The practical effect is field-specificity. A score of 6 on a computational biology paper is anchored to eLife's blinding and biological-replicate standards. A 6 on a machine learning paper is anchored to ICLR's iso-compute baseline requirements. The corpus is large enough to surface field-specific examples, not a single generic "peer review" archetype.

For a deeper look at how the corpus is built and maintained, see how our eight agents were calibrated on 69,000 real peer reviews.

Eight agents scoring independently

The eight agents each evaluate a different facet of the paper:

1. Methodology -- experimental design, statistics, controls, platform-specific checklists (CONSORT, PRISMA, NeurIPS Reproducibility Checklist, STROBE)
2. Formulas & Equations -- mathematical correctness, derivations, unit consistency
3. Originality -- textual overlap, self-plagiarism signals, integrity red flags (calibrated against PubPeer forensic patterns and OpenReview dual-submission catches)
4. Literature Coverage -- citation completeness, engagement with recent and relevant prior work
5. Reproducibility -- code availability, dataset access, version reporting, protocol detail
6. Clarity & Language -- argument structure, coherence, writing quality
7. Figures & Tables -- visual integrity, completeness, error bars, statistical reporting in figures
8. Prior Publication -- automated search across seven external sources (CrossRef, PubMed, arXiv, bioRxiv, medRxiv, Europe PMC and Unpaywall) plus our 4.5M-paper local library

The Prior Publication agent is not a language model. It runs parallel queries against those eight sources and returns a structured match report. A strong match is flagged in the report; the agent's own score is shown separately and is not averaged into the overall score.

The remaining seven agents each produce a 1-10 score without seeing each other's output. The overall score is their weighted average, with Methodology weighted most; a synthesis step then writes the summary. If any agent scores 2 or lower, the overall score is capped at 4.0, so a well-written paper with a disqualifying flaw cannot average its way into Minor revision.

What your score means for next steps

The overall score provides a directional signal calibrated to the real-world standards encoded in the corpus:

• Below 4.0: Reject. Significant methodological or integrity concerns. The per-agent reports identify which sections, figures, or statistical choices require attention before this paper can be submitted anywhere.
• 4.0 to 6.4: Major revision. The report is actionable and specific -- it cites section numbers, figures, and missing controls.
• 6.5 to 8.4: Minor revision.
• 8.5 and above: Accept.

What this means for authors

Our Pre-Check estimates acceptance from your title and abstract in seconds; the full eight-agent AI Review reads the whole PDF.

If you want to situate your contribution before the draft is final, our research-gap finder lets you search a topic free across the 4.5M-paper local library, where 120,000+ open research gaps have been mined.

When you are ready, run a full AI Review on the PDF. All eight agents run in parallel; the complete report typically arrives in under 15 minutes.

Frequently asked questions

Do all eight agents contribute equally to the overall score?

No. The overall score is a weighted average of the agents' scores. Methodology and Originality carry more weight than Clarity and Language, because methodological and integrity failures are disqualifying. A poorly written paper can be revised; a fundamental design flaw or plagiarism signal cannot.

Can a paper score differently on two separate runs?

Yes. Language models have inherent variation, so individual agent scores can differ between runs on the same paper.

Is the score comparable across disciplines?

Partially. The corpus covers 10 disciplines (Engineering, Life Sciences, Physical Sciences, Computer Science, Medicine, Mathematics, Environmental Science, Social Science, Humanities, Economics), so discipline-specific standards are represented. A 6 on a clinical trial is calibrated against CONSORT requirements; a 6 on a machine learning paper is calibrated against ICLR reproducibility criteria. Cross-discipline comparisons should be made with that context in mind.

What does the Prior Publication agent actually report?

It searches seven external sources (CrossRef, PubMed, arXiv, bioRxiv, medRxiv, Europe PMC and Unpaywall) and our 4.5M-paper local library for title and abstract matches. A strong match flags the paper for closer review and appears as a structured note in the report. If you have posted a preprint, that is expected and the agent distinguishes between a prior preprint and a prior full publication.

How is the 69,000-review corpus kept current?

Scrapers run on a scheduled basis, updating the corpus as new open peer reviews are published. Reviews used for calibration come from platforms that publish their peer reviews openly -- OpenReview, eLife, SciPost, PLOS ONE, and others. The retrieval layer queries the updated corpus on each review run, so new examples become available without redeploying the model.

AI Review reports are written by AI agents. They are pre-submission feedback, not an editorial decision.