When not to use AI peer review: honest limits and edge cases

Updated 26 October 2026. This post was written when AI Review ran eight agents. Since 25 October 2026 it runs nine: a Research Gap agent now checks the gap a manuscript claims against comparable published papers. Seven of the eight language-model agents are calibrated on the peer-review corpus; the Research Gap agent reads published papers instead, and Prior Publication is a lookup.

Science AI Journal reviews papers using nine AI agents, seven of them calibrated against an index of 5,000+ real peer-review texts from 15 open-review platforms. The corpus spans many fields, but it is not infinite, and some papers fall outside what the system can evaluate with confidence. Understanding where those edges are is part of using any tool well.

Why this question matters

No review system -- human or AI -- is equally strong across every possible paper. A statistician reviewing a pure philosophy paper is out of their depth. A clinical trialist assigned to a machine learning paper is not much better. The difference with AI is that its inputs are traceable: we can tell you which journals and platforms each agent's rubric is written around, and which platforms its calibration examples come from, and therefore where its review norms are well covered and where they are thin.

That transparency is useful. A 7 out of 10 from the methodology agent means something when the paper is a machine learning paper: its rubric carries the NeurIPS, ICLR and ICML checklists, and OpenReview, which holds about 54% of the index's peer-review texts, supplied 19 of its 60 calibration examples on 28 October 2026. It means something different when the paper is a study in computational folkloristics with no comparable open peer-review record.

Where the rubrics are strongest

The agents draw their rubrics from several platform clusters, written into each agent's prompt in journal/src/components/hakem/ScienceAI.tsx. Each cluster covers a study-type template with field-specific criteria:

• Machine learning and AI. OpenReview (NeurIPS, ICLR, ICML, AISTATS, TMLR, EMNLP, ACL, COLM, MIDL, CoRL, CVPR) forms the largest single block. The agents know what NeurIPS calls seed averaging, what ICLR means by iso-compute baselines, and what counts as data leakage in a benchmark setting.
• Life sciences. eLife, Nature Communications, PLOS ONE, F1000Research. The methodology agent knows eLife's biological-replicate rule (n = 3 technical replicates is not n = 3 biological replicates), Nature Communications' IRB requirements, and the power-analysis expectation at eLife.
• Clinical research. BMJ Open, F1000Research, Open Research Europe. The methodology agent checks CONSORT 2010 (randomized trials), STROBE (observational), and PRISMA (systematic reviews).
• Physics and mathematics. SciPost, Copernicus Publications. The rubrics tell the agents to flag the conjecture-vs-proof distinction that SciPost referees enforce, and to require ensemble uncertainty for Copernicus climate projections.
• Ecology and evolutionary biology. PCI Ecology, PCI Evolutionary Biology, PCI Mathematical & Computational Biology, PCI Neuroscience. Detection probability in occupancy models, phylogenetic autocorrelation in comparative analyses, pseudo-absence selection in species distribution models.

Within these fields, a score band rests on the field's own checklist, not on an abstract impression of quality. The agents' calibration examples are another matter: they are chosen by each agent's job, not by your field, and on 28 October 2026 they came only from OpenReview, SciPost, PeerJ and JOSS, for every paper. Our complete guide to AI peer review explains how each agent's rubric was derived from these sources.

Where the rubrics are thinner

Not every field has an open peer-review tradition. Some publish referee reports only on request; others keep reviews confidential by default. This leaves real gaps.

Humanities and interpretive disciplines. We review humanities manuscripts like any other. The clarity, literature-coverage, and originality agents are useful regardless of field. But the methodology rubric was calibrated primarily on empirical and formal-science papers. For a hermeneutic or archival paper, the "study design" and "statistical methods" sections of the methodology report are not the right frame. Read those sections with that context in mind.

Highly specialized engineering subfields. Industrial process engineering, certain materials characterization methods, and proprietary manufacturing workflows sometimes have no comparable public peer-review record. The agents can evaluate structure, citation integrity, and reproducibility language -- but the domain-specific criterion for what counts as a sufficient materials characterization protocol may not be in the calibration data.

Economics and quantitative social science. No rubric cluster is written for these fields and no calibration example comes from a social-science venue, so the agents apply their general checks. An econometrics paper with a novel instrumental-variables design will receive a reasonable structural review; the depth of domain-specific critique is lower.

Genuinely novel interdisciplinary work. Papers defined by crossing field boundaries -- applying astrophysics methods to biological data, for example -- may fall between the venue categories the agents' rubrics are written for. The review is still useful as a structured checklist, but interpret the domain-specific commentary with that limitation in mind.

What the agents cannot do regardless of field

Some limits are not about calibration depth but about what a language-model agent can do at inference time.

Run code. The reproducibility agent evaluates whether code is available, documented, and described with version numbers and random seeds. It cannot execute the code to verify that it actually produces the reported results. If the central claim depends on running a non-trivial algorithm correctly, the reproducibility score is a proxy for good practice, not a verification of the result.

Access external resources. Agents receive the manuscript text. They do not follow URLs, download data from Zenodo or OSF, or read supplementary files hosted elsewhere. The figures agent evaluates figures embedded in the manuscript; supplementary panels posted on external platforms are invisible to it.

Independently verify a novel proof. The methodology agent is calibrated to flag cases where a proof-by-example is presented as a general result, or where "we show X" is used for what is actually a conjecture -- patterns SciPost referees enforce. It cannot independently prove or disprove a new theorem. For a paper whose central contribution is a novel proof, treat the mathematical assessment as a structural and framing check, not a validity check. Our hallucination guardrails post covers how the system signals uncertainty rather than confabulating a verdict.

Detect prior publication in paywalled databases. The prior-publication agent checks CrossRef, PubMed, Europe PMC, Unpaywall, arXiv, medRxiv, bioRxiv, and our local 7.9M-paper FTS5 index (openalex.org metadata informs much of that index). It does not have access to publisher internal systems, papers behind paywalls without CrossRef or Unpaywall metadata, or double-submission records at other journals. A preprint on arXiv is very likely to be found; a paper published only in a paywalled journal without CrossRef indexing may not be.

What to do if your paper is near the edge

Before running a full review, a Pre-Check is a quick way to see where your paper sits. It reads only the title and abstract and returns, in seconds, the field it detects and a ranked shortlist of journals where similar papers were published. If the detected field is wrong, or few comparable papers come back, treat that as a sign your topic sits near the edge of what we index. In the full review itself, if the flags land in areas with strong calibration -- methodology for a life-sciences paper, reproducibility for a machine learning paper -- take them seriously. If the flags are in areas you suspect sit at the edge of our corpus, treat the score as directional: it tells you whether the paper follows open-science norms, not whether the domain-specific contribution is significant.

If you are uncertain, the review is still a useful structured checklist for the non-domain-specific items: citation integrity, figure quality, reproducibility language, and prior-publication signals. Those checks add value regardless of calibration depth in your specific subfield.

The research-gap finder can also give you a signal about whether your topic sits in a well-indexed area of our corpus. If there are synthesized gaps in your field, the literature-coverage and originality agents are more likely to be working from a populated reference distribution.

What this means for authors

A review from Science AI Journal is most reliable when your paper maps clearly to one of the venue clusters its rubrics are written for: machine learning, life sciences, clinical research, physics, or ecology. It is still useful for other fields, but interpret the domain-specific commentary with the calibration context in mind.

For questions about fit, the fastest path is a Pre-Check -- it returns a result in seconds. If the field it detects and the journals it suggests look right for your work, that is a good first sign; the calibration notes above tell you how much weight to give the full review's domain-specific comments. When the paper is clearly within our coverage, run the full AI Review.

AI Review reports are written by AI agents. They are pre-submission feedback, not an editorial decision.

Frequently asked questions

My paper proves a new theorem. Can the AI review it?

Yes, with qualifications. The originality, literature-coverage, and clarity agents will give useful feedback regardless of field. The methodology agent is calibrated to flag the conjecture-vs-proof distinction that SciPost referees enforce -- a real and common issue. It cannot, however, independently verify a new proof. Treat the mathematical assessment as a structural and framing check, not a validity check.

Can the prior-publication agent miss a published version of my paper?

It can. We check CrossRef, PubMed, Europe PMC, Unpaywall, arXiv, medRxiv, bioRxiv, and a local FTS5 index of 7.9M+ papers -- but not every journal's internal database or papers that exist only behind paywalls without CrossRef or Unpaywall metadata. A preprint on arXiv is very likely to be found; a paper in a paywalled journal with no CrossRef listing may not be.

Which fields are best covered by your calibration data?

The rubrics are written for machine learning and AI, life sciences, clinical research, physics and mathematics, and ecology and evolutionary biology. The examples the seven calibrated agents are shown come from fewer fields: on 28 October 2026 every one was from OpenReview (machine learning), SciPost (physics and mathematics), PeerJ (life sciences) or JOSS (research software).

What should I do if my paper is highly specialized and does not fit standard templates?

Run the full AI Review and lean on its structural checks -- citation integrity, reproducibility language, figure labeling, prior-publication signals -- which are useful regardless of field specificity. If the domain-specific flags look like a poor fit, focus on what the review does tell you and note the calibration limitation when you share the report.

Does the system work for humanities papers?

The clarity, originality, and literature-coverage agents produce useful feedback for humanities work. The methodology agent's study-type templates (machine learning, life sciences, clinical, physics, ecology) are not designed for interpretive or archival research. For humanities papers, the methodology score is a rough structural check rather than a domain-calibrated evaluation.