Hallucination Guardrails for an AI Reviewer: What We Do, What We Still Can't Fix

The first question researchers ask about AI peer review is not "how fast?" It is: can it hallucinate? Can the system invent a methodological flaw that is not in the paper, cite a statistical principle that does not apply, or flag a missing reference to a work the agent cannot actually retrieve? These are the right questions. Here is what the guardrails look like, and where they stop.

Why hallucination is the central risk in AI review

Unlike an AI writing assistant or a chatbot, an AI reviewer makes specific, consequential claims about a specific document. It says "Section 3.2 lacks a power analysis," not "power analysis is generally important." That specificity is what makes it useful, and it is also what makes fabrication costly. A vague hallucination in a chatbot response is annoying. A fabricated methodological charge in a peer review report, one that the paper's authors cannot find in their own manuscript, wastes time and erodes trust.

The design choice we made early: specificity is enforced at the prompt level, not at the post-processing level.

Guard one: every claim must cite the manuscript

Every language-model agent carries this requirement in its own prompt. The methodology agent's wording:

Every concern MUST include a direct quote or specific reference from the manuscript (e.g., "Methods §2.1 states 'n=45 per arm'" or "Table 3 reports p<0.05 without exact values"). Concerns without manuscript evidence are unacceptable.

The formulas agent goes one step further: every concern must quote the specific equation or formula it is about.

This does not prevent all hallucinations. A model can hallucinate a specific section number as easily as a vague statement. But it changes the failure mode in a useful way. When the claim is anchored to "Section 3.2," the author can go to Section 3.2 and check whether the concern is real. That checkability is the practical benefit: a groundless claim is immediately falsifiable, not merely dismissible.

Guard two: the citations the literature agent names are checked against the manuscript

One of the most common AI failure modes in review contexts is the confident "missing citation" claim: "the authors neglect to cite Smith et al. 2019, which directly contradicts their argument." An agent with no access to external literature is fabricating that confidence.

Our literature-coverage agent is told to ground its evaluation in what the input actually contains, and when the manuscript shows only citation keys rather than a bibliography, its prompt draws a hard line:

When discussing missing citations, describe gaps in CATEGORIES (e.g., "no foundational uniqueness theory is cited") — do NOT name specific authors, papers, years, or DOIs as missing.

The prompt is not trusted on its own. After the agent answers, a deterministic verifier -- no language model involved -- checks every author-year citation, DOI and citation key the agent names against the manuscript, and any that do not appear there are shown to you, flagged as potentially fabricated. The agent evaluates the citations that are present: their diversity, their currency, whether they support the claims they are attached to, and whether self-citation rates look unusual.

Guard three: structured JSON-only output

Each agent returns its response in a fixed JSON schema. There is no free-form narrative that can drift into invented content between structured sections. The synthesis step aggregates per-agent JSON into the final report. A parse failure is flagged, not hidden: the agent's card is marked as a fallback, its score is labelled a neutral placeholder rather than an assessment, and the report suggests a re-run.

This matters because the alternative -- accepting whatever text the model produces and presenting it to the author -- creates a fallback where "no valid response" looks identical to "a valid negative response." Structured output with explicit failure handling means that if something goes wrong, the author sees that an agent did not complete, not a hallucinated verdict.

Guard four: prior publication detection uses database lookup, not LLM inference

The prior publication agent does not ask the language model whether a paper has been published before. It fans out to seven external sources -- CrossRef, PubMed, Europe PMC, Unpaywall, arXiv, medRxiv and bioRxiv -- and to our local 4.5M-paper library, querying each for title and abstract overlap. High-confidence matches use a 60%+ word-overlap scoring function, not language model judgement.

The local library FTS5 index uses BM25 ranking, the same retrieval algorithm that underlies most production search engines. The result is a list of candidate duplicates with overlap scores, not an AI's guess that "this might have been published." A detailed breakdown of how the detection works is in our prior publication detection post.

This separation matters because prior publication is the one check where a false positive has immediate, serious consequences. We want the detection to be auditable and based on database evidence, not model confidence.

Guard five: calibration examples anchor agents to documented review norms

Before producing any output, each language-model agent retrieves up to 60 examples chosen for its specialty from a corpus of 69,000+ real peer reviews from OpenReview, eLife, SciPost, BMJ Open, PLOS ONE, PCI, and 13 other platforms. The corpus itself is described on our dataset page. The examples demonstrate what "a major concern" looks like in practice, in the vocabulary of real reviewers doing that agent's job.

This is a form of retrieval-augmented generation, as described by Lewis et al. in "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks" (arXiv:2005.11401, 2020). Our application differs from the canonical setup in that the retrieved documents are not background facts but worked examples of expert judgement.

Calibration does not eliminate hallucination, but it reduces the probability of inventing concepts that the relevant community would not recognize. An agent calibrated against 11,433 SciPost reviews has seen what a SciPost referee means by "this is a conjecture, not a theorem." An agent pre-trained only on general text has to guess.

Where the guardrails stop

Citation-level plagiarism detection. The originality agent works from the manuscript text alone. It can flag text-reuse signals, statistical inconsistencies that are arithmetically verifiable from the reported numbers (such as GRIM test failures and p-values inconsistent with their stated test statistics), and duplicate-submission patterns in the metadata. It cannot detect verbatim copying from a paper that is not mentioned anywhere in the manuscript. That check requires an external database query, which is out of scope for the text-analysis agent.

External literature verification. The literature agent can evaluate the citations that are present in the paper. It cannot confirm that a cited paper says what the authors claim it says. Misrepresentation of a cited source is a failure mode that requires reading the cited paper, which the agent does not do.

Platform coverage gaps. The calibration corpus skews toward machine learning and computer science, where open peer review is most common. OpenReview accounts for nearly half of the 69,000+ records. Niche fields -- specialized chemistry subdisciplines, for example, or narrow engineering domains -- may have review norms not well represented in any of the 19+ platforms we cover. Agents calibrated on this corpus may be less precisely calibrated for those fields.

The agent cannot fix a flawed manuscript. This is not a hallucination risk, but it is the limit researchers most often underestimate. The report is a diagnosis. If the methodology agent identifies that Section 3.2 has no power analysis, that is accurate. What to do about it is the author's work, exactly as it would be with a human reviewer's letter.

What this means for authors

When you read an AI review from our system, the right way to evaluate a concern is the same as with a human review: go to the section it cites and check whether the concern is valid. If the citation is wrong -- if "Section 3.2" does not exist, or the concern does not match what is there -- that is a detectable error you can contest. If the concern is real, the manuscript-grounded framing gives you a specific location to address in your revision.

You can run AI Review on the full manuscript before submitting to any journal; its report carries the specificity requirements and structured output described above. For a faster first read of fit from the title and abstract alone, there is Pre-Check. For identifying open questions in your field before you start writing, the research-gap finder holds 120,000+ open gaps mined from 100,000+ papers in our local library, and searching a topic there is free.

Frequently asked questions

Can the AI invent a statistic that is not in the paper?

The specificity requirement instructs the agent to cite the section, figure, or table containing the statistic it is discussing. A fabricated statistic would need a fabricated section reference, which the author can immediately check. This makes hallucinated numeric claims auditable rather than impossible, which is the realistic standard.

How does the system handle a case where the agent cannot find a valid concern?

The rubric allows agents to score highly and list no critical issues. "None identified" is a valid and common output. The structured format has an explicit "none identified" state for each concern section; it does not require the agent to produce concerns to fill a template.

Does the prior publication check access my full text?

The prior publication check queries external databases and the local library using the title and abstract you submit. The full text is not sent to CrossRef, Unpaywall, arXiv, medRxiv, bioRxiv, or the FTS5 index. Match scores are computed on title and abstract overlap only.

What happens if an agent produces a response that fails the JSON schema?

The system flags it. That agent's card is marked as a fallback, and its score is labelled a neutral placeholder, not an assessment -- you see that the agent did not complete, rather than a clean report it never wrote. Re-running the review usually produces the full result.

Is the specificity requirement enforced after the model responds, or before?

It is a prompt-level instruction, not post-processing. The model is instructed to include manuscript references before it generates any text. We do not strip or annotate output to add references that were not produced. If the model ignores the instruction, the output will read as vague, and a vague concern is easier for an author to challenge than a falsely specific one.

AI Review reports are written by AI agents. They are pre-submission feedback, not an editorial decision.