When not to use AI peer review: honest limits and edge cases

Science AI Journal reviews papers using eight AI agents calibrated on 69,000 real peer reviews from 19 platforms. That is a wide corpus, and the coverage is genuinely broad. But it is not infinite, and some papers fall outside what the system can evaluate with confidence. Understanding where those edges are is part of using any tool well.

Why this question matters

No review system -- human or AI -- is equally strong across every possible paper. A statistician reviewing a pure philosophy paper is out of their depth. A clinical trialist assigned to a machine learning paper is not much better. The difference with AI is that the calibration data is traceable: we can tell you exactly which journals and platforms informed each agent's rubric, and therefore where the reference distribution is dense and where it is thin.

That transparency is useful. A 7 out of 10 from the methodology agent means something when the paper is a machine learning paper and the agent is calibrated on 33,469 OpenReview decisions from NeurIPS, ICLR, ICML, AISTATS, and TMLR. It means something different when the paper is a study in computational folkloristics with no comparable open peer-review record.

Where calibration is strong

The agents draw their rubrics from several platform clusters, documented in journal/src/lib/reviewers/prompts.ts. Each cluster covers a study-type template with field-specific criteria:

• Machine learning and AI. OpenReview (NeurIPS, ICLR, ICML, AISTATS, TMLR, EMNLP, ACL, COLM, MIDL, CoRL, CVPR) forms the largest single block. The agents know what NeurIPS calls seed averaging, what ICLR means by iso-compute baselines, and what counts as data leakage in a benchmark setting.
• Life sciences. eLife, Nature Communications, PLOS ONE, F1000Research. The methodology agent knows eLife's biological-replicate rule (n = 3 technical replicates is not n = 3 biological replicates), Nature Communications' IRB requirements, and the power-analysis expectation at eLife.
• Clinical research. BMJ Open, F1000Research, Open Research Europe. The methodology agent checks CONSORT 2010 (randomized trials), STROBE (observational), and PRISMA (systematic reviews), item by item.
• Physics and mathematics. SciPost, Copernicus Publications. The agents are calibrated to flag the conjecture-vs-proof distinction that SciPost referees enforce, and to require ensemble uncertainty for Copernicus climate projections.
• Ecology and evolutionary biology. PCI Ecology, PCI Evolutionary Biology, PCI Mathematical & Computational Biology, PCI Neuroscience. Detection probability in occupancy models, phylogenetic autocorrelation in comparative analyses, pseudo-absence selection in species distribution models.

Within these fields, a score band reflects a well-populated reference distribution, not an abstract impression of quality. Our complete guide to AI peer review explains how each agent's rubric was derived from these sources.

Where calibration is thinner

Not every field has an open peer-review tradition. Some publish referee reports only on request; others keep reviews confidential by default. This leaves real gaps.

Humanities and interpretive disciplines. We cover Humanities as one of our 10 disciplines. The clarity, literature-coverage, and originality agents are useful regardless of field. But the methodology rubric was calibrated primarily on empirical and formal-science papers. For a hermeneutic or archival paper, the "study design" and "statistical methods" sections of the methodology report are not the right frame. Read those sections with that context in mind.

Highly specialized engineering subfields. Industrial process engineering, certain materials characterization methods, and proprietary manufacturing workflows sometimes have no comparable public peer-review record. The agents can evaluate structure, citation integrity, and reproducibility language -- but the domain-specific criterion for what counts as a sufficient materials characterization protocol may not be in the calibration data.

Economics and quantitative social science. Coverage exists -- the methodology and literature agents are calibrated on some social-science venues -- but it is thinner than for machine learning or life sciences. An econometrics paper with a novel instrumental-variables design will receive a reasonable structural review; the depth of domain-specific critique is lower.

Genuinely novel interdisciplinary work. Papers defined by crossing field boundaries -- applying astrophysics methods to biological data, for example -- may fall between the venue categories the agents were calibrated against. The review is still useful as a structured checklist, but interpret the domain-specific commentary with that limitation in mind.

What the agents cannot do regardless of field

Some limits are not about calibration depth but about what a language-model agent can do at inference time.

Run code. The reproducibility agent evaluates whether code is available, documented, and described with version numbers and random seeds. It cannot execute the code to verify that it actually produces the reported results. If the central claim depends on running a non-trivial algorithm correctly, the reproducibility score is a proxy for good practice, not a verification of the result.

Access external resources. Agents receive the manuscript text. They do not follow URLs, download data from Zenodo or OSF, or read supplementary files hosted elsewhere. The figures agent evaluates figures embedded in the manuscript; supplementary panels posted on external platforms are invisible to it.

Independently verify a novel proof. The methodology agent is calibrated to flag cases where a proof-by-example is presented as a general result, or where "we show X" is used for what is actually a conjecture -- patterns SciPost referees enforce. It cannot independently prove or disprove a new theorem. For a paper whose central contribution is a novel proof, treat the mathematical assessment as a structural and framing check, not a validity check. Our hallucination guardrails post covers how the system signals uncertainty rather than confabulating a verdict.

Detect prior publication in paywalled databases. The prior-publication agent checks six sources: CrossRef, Unpaywall, arXiv, medRxiv, bioRxiv, and our local 4.4M-paper FTS5 index (openalex.org metadata informs much of that index). It does not have access to publisher internal systems, papers behind paywalls without CrossRef or Unpaywall metadata, or double-submission records at other journals. A preprint on arXiv is very likely to be found; a paper published only in a paywalled journal without CrossRef indexing may not be.

What to do if your paper is near the edge

Use the pre-check scorer before running a full review. The scorer returns a structured signal across the same eight dimensions. If the flags land in areas with strong calibration -- methodology for a life-sciences paper, reproducibility for a machine learning paper -- take them seriously. If the flags are in areas you suspect sit at the edge of our corpus, treat the score as directional: it tells you whether the paper follows open-science norms, not whether the domain-specific contribution is significant.

If you are uncertain, the review is still a useful structured checklist for the non-domain-specific items: citation integrity, figure quality, reproducibility language, and prior-publication signals. Those checks add value regardless of calibration depth in your specific subfield.

The research-gap finder can also give you a signal about whether your topic sits in a well-indexed area of our corpus. If there are synthesized gaps in your field, the literature-coverage and originality agents are more likely to be working from a populated reference distribution.

What this means for authors

A review from Science AI Journal is most reliable when your paper maps clearly to one of the well-calibrated venue clusters: machine learning, life sciences, clinical research, physics, or ecology. It is still useful for other fields, but interpret the domain-specific commentary with the calibration context in mind.

For questions about fit, the fastest path is to run the pre-check scorer -- it returns a result in under a minute. If the score and flags look meaningful for your field, the full review will be, too. When the paper is clearly within our coverage, submit for a full AI peer review.

Reviews are performed by AI agents. Editorial decisions are made by human editors.

Frequently asked questions

My paper proves a new theorem. Can the AI review it?

Yes, with qualifications. The originality, literature-coverage, and clarity agents will give useful feedback regardless of field. The methodology agent is calibrated to flag the conjecture-vs-proof distinction that SciPost referees enforce -- a real and common issue. It cannot, however, independently verify a new proof. Treat the mathematical assessment as a structural and framing check, not a validity check.

Can the prior-publication agent miss a published version of my paper?

It can. We check six sources -- CrossRef, Unpaywall, arXiv, medRxiv, bioRxiv, and a local FTS5 index of 4.4M+ papers -- but not every journal's internal database or papers that exist only behind paywalls without CrossRef or Unpaywall metadata. A preprint on arXiv is very likely to be found; a paper in a paywalled journal with no CrossRef listing may not be.

Which fields are best covered by your calibration data?

Machine learning and AI, life sciences, clinical research, physics and mathematics, and ecology and evolutionary biology. Within those, score bands reflect documented journal-decision distributions, not abstract quality impressions.

What should I do if my paper is highly specialized and does not fit standard templates?

Run the pre-check scorer first. The structural checks -- citation integrity, reproducibility language, figure labeling, prior-publication signals -- are useful regardless of field specificity. If the domain-specific flags look like a poor fit, focus on what the review does tell you and note the calibration limitation when you share the report.

Does the system work for humanities papers?

The clarity, originality, and literature-coverage agents produce useful feedback for humanities work. The methodology agent's study-type templates (machine learning, life sciences, clinical, physics, ecology) are not designed for interpretive or archival research. For humanities papers, the methodology score is a rough structural check rather than a domain-calibrated evaluation.