Hallucination Guardrails for an AI Reviewer: What We Do, What We Still Can't Fix

The first question researchers ask about AI peer review is not "how fast?" It is: can it hallucinate? Can the system invent a methodological flaw that is not in the paper, cite a statistical principle that does not apply, or flag a missing reference to a work the agent cannot actually retrieve? These are the right questions. Here is what the guardrails look like, and where they stop.

Why hallucination is the central risk in AI review

Unlike an AI writing assistant or a chatbot, an AI reviewer makes specific, consequential claims about a specific document. It says "Section 3.2 lacks a power analysis," not "power analysis is generally important." That specificity is what makes it useful, and it is also what makes fabrication costly. A vague hallucination in a chatbot response is annoying. A fabricated methodological charge in a peer review report, one that the paper's authors cannot find in their own manuscript, wastes time and erodes trust.

The design choice we made early: specificity is enforced at the prompt level, not at the post-processing level.

Guard one: every claim must cite the manuscript

The instruction to each of the eight agents includes this requirement, embedded directly in each agent's system prompt:

Every strength and concern you report MUST cite a specific section number, figure, table, equation, or direct quote from the manuscript. Generic statements like "the methodology is weak" are unacceptable -- instead write "Section 3.2 lacks a power analysis for the sample size of n=42" or "Figure 3 has no error bars despite reporting mean values." Concerns without specific manuscript references will be discarded.

The same rule applies to the questions each agent poses to the author:

Each question MUST cite a specific section/figure/table and, where applicable, the missing parameter or control.

This does not prevent all hallucinations. A model can hallucinate a specific section number as easily as a vague statement. But it changes the failure mode in a useful way. When the claim is anchored to "Section 3.2," the author can go to Section 3.2 and check whether the concern is real. That checkability is the practical benefit: a groundless claim is immediately falsifiable, not merely dismissible.

Guard two: the literature agent is forbidden from speculating about absent citations

One of the most common AI failure modes in review contexts is the confident "missing citation" claim: "the authors neglect to cite Smith et al. 2019, which directly contradicts their argument." An agent with no access to external literature is fabricating that confidence.

Our literature-coverage agent's system prompt includes an explicit prohibition:

Do NOT speculate about citations that "should have been included" -- you cannot access external literature to judge absence.

The agent evaluates the citations that are present in the manuscript: their diversity, their currency, whether they support the claims they are attached to, and whether self-citation rates look unusual. It does not invent a hidden canon of missing works it cannot actually retrieve.

Guard three: structured JSON-only output

Each agent returns its response in a fixed JSON schema. There is no free-form narrative that can drift into invented content between structured sections. The synthesis step aggregates per-agent JSON into the final report. A parse failure is treated as a failed gate, not a blank default: the system raises an explicit parseFailed flag rather than silently returning an empty or neutral score.

This matters because the alternative -- accepting whatever text the model produces and presenting it to the author -- creates a fallback where "no valid response" looks identical to "a valid negative response." Structured output with explicit failure handling means that if something goes wrong, the author sees a system failure, not a hallucinated verdict.

Guard four: prior publication detection uses database lookup, not LLM inference

The prior publication agent does not ask the language model whether a paper has been published before. It fans out to six external sources -- CrossRef, Unpaywall, arXiv, medRxiv, bioRxiv, and the local 4.4M-paper institutional library -- querying each for title and abstract overlap. High-confidence matches use a 60%+ word-overlap scoring function, not language model judgement.

The local library FTS5 index uses BM25 ranking, the same retrieval algorithm that underlies most production search engines. The result is a list of candidate duplicates with overlap scores, not an AI's guess that "this might have been published." A detailed breakdown of how the detection works is in our prior publication detection post.

This separation matters because prior publication is the one check where a false positive has immediate, serious consequences. We want the detection to be auditable and based on database evidence, not model confidence.

Guard five: calibration examples anchor agents to documented review norms

Before producing any output, each agent retrieves up to 50 matched examples from a corpus of 69,000+ real peer reviews from OpenReview, eLife, SciPost, BMJ Open, PLOS ONE, PCI, and 11 other platforms. Full details are in our calibration post. The examples demonstrate what "a major concern" looks like in practice, in the vocabulary of real reviewers, for papers similar to the one under review.

This is a form of retrieval-augmented generation, as described by Lewis et al. in "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks" (arXiv:2005.11401, 2020). Our application differs from the canonical setup in that the retrieved documents are not background facts but worked examples of expert judgement.

Calibration does not eliminate hallucination, but it reduces the probability of inventing concepts that the relevant community would not recognize. An agent calibrated against 11,433 SciPost reviews has seen what a SciPost referee means by "this is a conjecture, not a theorem." An agent pre-trained only on general text has to guess.

Where the guardrails stop

Citation-level plagiarism detection. The originality agent works from the manuscript text alone. It can flag text-reuse signals, statistical inconsistencies that are arithmetically verifiable from the reported numbers (such as GRIM test failures and p-values inconsistent with their stated test statistics), and duplicate-submission patterns in the metadata. It cannot detect verbatim copying from a paper that is not mentioned anywhere in the manuscript. That check requires an external database query, which is out of scope for the text-analysis agent.

External literature verification. The literature agent can evaluate the citations that are present in the paper. It cannot confirm that a cited paper says what the authors claim it says. Misrepresentation of a cited source is a failure mode that requires reading the cited paper, which the agent does not do.

Platform coverage gaps. The calibration corpus skews toward machine learning and computer science, where open peer review is most common. OpenReview accounts for roughly 48% of the 69,000+ records. Niche fields -- specialized chemistry subdisciplines, for example, or narrow engineering domains -- may have review norms not well represented in any of the 15 platforms we cover. Agents calibrated on this corpus may be less precisely calibrated for those fields.

The agent cannot fix a flawed manuscript. This is not a hallucination risk, but it is the limit researchers most often underestimate. The report is a diagnosis. If the methodology agent identifies that Section 3.2 has no power analysis, that is accurate. What to do about it is the author's work, exactly as it would be with a human reviewer's letter.

What this means for authors

When you read an AI review from our system, the right way to evaluate a concern is the same as with a human review: go to the section it cites and check whether the concern is valid. If the citation is wrong -- if "Section 3.2" does not exist, or the concern does not match what is there -- that is a detectable error you can contest. If the concern is real, the manuscript-grounded framing gives you a specific location to address in your revision.

You can run a free pre-submission check at our scorer before submitting to any journal. The report structure is the same as a full review, including the specificity requirements and structured output described above. For identifying open questions in your field before you start writing, the research-gap finder surfaces gaps drawn from 250M+ papers indexed via OpenAlex.

Frequently asked questions

Can the AI invent a statistic that is not in the paper?

The specificity requirement instructs the agent to cite the section, figure, or table containing the statistic it is discussing. A fabricated statistic would need a fabricated section reference, which the author can immediately check. This makes hallucinated numeric claims auditable rather than impossible, which is the realistic standard.

How does the system handle a case where the agent cannot find a valid concern?

The rubric allows agents to score highly and list no critical issues. "None identified" is a valid and common output. The structured format has an explicit "none identified" state for each concern section; it does not require the agent to produce concerns to fill a template.

Does the prior publication check access my full text?

The prior publication check queries external databases and the local library using the title and abstract you submit. The full text is not sent to CrossRef, Unpaywall, arXiv, medRxiv, bioRxiv, or the FTS5 index. Match scores are computed on title and abstract overlap only.

What happens if an agent produces a response that fails the JSON schema?

The system raises an explicit failure flag and does not substitute a neutral default. You would see a system-level error on that agent rather than a fabricated clean report.

Is the specificity requirement enforced after the model responds, or before?

It is a prompt-level instruction, not post-processing. The model is instructed to include manuscript references before it generates any text. We do not strip or annotate output to add references that were not produced. If the model ignores the instruction, the output will read as vague, and a vague concern is easier for an author to challenge than a falsely specific one.

Reviews are performed by AI agents. Editorial decisions are made by human editors.