How AI enforces CONSORT, STROBE, and PRISMA in peer review

Reporting checklists exist because peer reviewers are inconsistent. A seasoned clinical trialist will spot a missing allocation concealment statement on page one; a reviewer from a neighbouring field may miss it entirely. CONSORT 2010 was designed to fix that inconsistency — to give every reader, and every reviewer, the same checklist. In practice, compliance is still voluntary and manual.

Our methodology agent applies that checklist automatically, to every paper, before it goes to a journal's editors.

Why reporting standards still fail in practice

CONSORT 2010 — the Consolidated Standards of Reporting Trials — was published in the BMJ after two decades of evidence that randomised controlled trials were systematically under-reported (Schulz KF et al., BMJ 2010, doi:10.1136/bmj.c332). The 25-item checklist covers everything from sequence generation to outcome switching. Studies since have linked journal endorsement of the checklist with more complete reporting of published trials.

Yet manual reviewer compliance remains inconsistent. Audits of published trials keep finding incomplete CONSORT reporting, including in journals whose instructions to authors require the checklist. The problem is not that reviewers disagree with the standard — it is that reviewing is cognitively demanding, and granular checklists are easy to skip when the scientific argument is compelling.

An AI agent does not make that trade-off. It applies the same checklist to paper 1 and paper 1,000.

What the methodology agent checks for CONSORT

When our methodology agent identifies a randomised controlled trial, it evaluates the paper against CONSORT 2010. Its rubric spells out the randomisation method, allocation concealment, blinding and outcome switching; alongside them, item 17 is one no trial report can skip:

Allocation concealment (item 9). The paper must describe the mechanism that concealed the allocation sequence — sealed envelopes, a central randomisation service, permuted-block via a pharmacy. Stating "patients were randomised 1:1" without naming the concealment mechanism fails item 9. It is a common omission in trials from authors unfamiliar with CONSORT.

Blinding (item 11). Three groups must be addressed: participants, those administering the intervention, and outcome assessors. Many papers blind participants and providers but say nothing about whether outcome assessors were blinded. The rubric asks for participant and assessor blinding to be stated, not implied.

Outcomes and estimation (item 17). CONSORT asks for the estimated effect size and its precision, such as a 95% confidence interval, for each primary and secondary outcome. A p-value alone does not meet item 17.

Outcome switching. The pre-specified primary outcome, in the Methods and in the trial registration, must be the outcome the Results section foregrounds. The rubric tells the agent to flag any paper that appears to have switched its primary outcome after data collection: post-hoc elevation of a secondary outcome is an integrity issue, not a minor revision.

Every concern the agent raises must quote or point to the manuscript text behind it, so authors know precisely what is missing, not just that something is wrong.

STROBE: what the agent checks for observational studies

The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement covers cohort, case-control, and cross-sectional studies. Three STROBE items the rubric singles out:

Exposure definition. The exposure must be operationally defined — what was measured, how, and when — not just labelled. "Smoking status" is a label. "Self-reported current smoker, defined as ≥1 cigarette per day for the 12 months prior to enrolment" is a definition.

Missing data mechanism. MCAR (missing completely at random), MAR (missing at random), or MNAR (missing not at random) must be stated, with the handling method justified. "We used complete-case analysis" is flagged unless the authors demonstrate the data are MCAR. Multiple imputation is expected when data are MAR.

Loss to follow-up. Loss exceeding 20% in a cohort study triggers a critical flag regardless of the reported results, because the completers may be systematically different from those lost. The threshold is written into the rubric's clinical-research criteria.

PRISMA: the agent's checks for systematic reviews

For systematic reviews and meta-analyses, the agent checks against PRISMA 2020. Its rubric names three checks:

PROSPERO registration. A systematic review without a PROSPERO registration number raises HARKing risk — the scope may have been adjusted after results were visible. The agent flags absent registration as a mandatory disclosure item.

Risk-of-bias assessment. Each included study's risk of bias has to be assessed and shown; the rubric asks for a QUADAS-2 or RoB 2 table. A review that pools studies without one leaves readers no way to weigh them.

GRADE certainty assessment. PRISMA 2020 asks for an assessment of certainty in the body of evidence for each outcome, usually made with GRADE. A systematic review that reports pooled estimates without any certainty assessment is flagged.

How calibration makes the check field-specific

The distinction between an agent that knows the CONSORT items and one that applies them as a real BMJ Open reviewer would is calibration. Before each review, our methodology agent receives up to 60 examples drawn from a corpus of 69,000+ real peer reviews using BM25 full-text search across our FTS5 index, with methodology search terms. Many come from platforms with explicit CONSORT, STROBE, or PRISMA compliance requirements, such as BMJ Open, PLOS ONE, F1000Research, Open Research Europe and the PCI networks.

The retrieval is matched to the agent's job, not to your paper's topic: the same methodology examples calibrate a drug trial and an ecological cohort study, and the rubric tells the agent which checklist applies. What the examples teach is the register of item-level specificity — "CONSORT item 9 (allocation concealment) is not addressed" rather than "methodology is incomplete."

The decision distribution of retrieved examples is also controlled: the retrieval aims for a mix of roughly 40% reject, 30% revision and 20% accept decisions, and it caps how many examples any single platform may supply. This prevents the agent from being over-calibrated to one venue's idiosyncratic standards. You can read more about the calibration corpus itself on our dataset page.

Score thresholds and what happens at each level

The methodology agent scores on a 1–10 scale, and its rubric fixes what each band means: 9–10 for a rigorous, reproducible design that is pre-registered or justifies why not; 7–8 for minor issues, such as a missing power analysis or one absent control; 5–6 for significant issues — an underpowered study, the wrong test, a missing key control — that call for major revision; 3–4 for fundamental flaws such as data leakage, causation claimed from correlation, or HARKing; and 1–2 for an irredeemably flawed design. A reporting gap lands in the band its consequences put it in: an unstated concealment method is not the same failure as a switched primary outcome. The score is not punitive: it is calibrated against real reviews from journals that publish under these standards.

The full report accompanying any score names what is missing, quotes the relevant manuscript section, and says what the authors need to supply. For authors who want to see this before submission, AI Review runs the full methodology check in under 15 minutes.

What this means for authors

If you are submitting a clinical trial, cohort study, or systematic review:

• Run your paper against the relevant checklist before submission. CONSORT, STROBE, and PRISMA are free at equator-network.org.
• Confirm your trial registration number appears in the Methods section — not just the abstract. The agent checks both.
• State explicitly whether outcome assessors were blinded. One sentence. Reviewers ask for it again and again, and it takes 30 seconds to add.
• For systematic reviews: if PROSPERO registration was omitted, explain when the protocol was fixed and why registration was deferred. The agent will flag absence; reviewers are more lenient when authors acknowledge the gap proactively.

AI Review runs the full methodology check, including CONSORT/STROBE/PRISMA compliance, in under 15 minutes and returns a report with item-level specificity. If you are still developing your research question, the research-gap finder can help position your study against the existing literature before you design your methods.

FAQ

Does the methodology agent check CONSORT for all study types?

No. The agent identifies the study type from the manuscript before applying a checklist. CONSORT applies to randomised trials. STROBE applies to cohort, case-control, and cross-sectional designs. PRISMA applies to systematic reviews and meta-analyses. For ML papers, the agent checks against NeurIPS/ICLR reproducibility standards. For theoretical physics or mathematics, it checks SciPost-style proof completeness.

What if my trial was registered but after enrolment began?

Retrospective registration is flagged as a significant methodological concern, but not an automatic rejection. The agent will note the gap between registration date and enrolment start date if you disclose it. Stating only the registration number without its date is the pattern to avoid: the registry shows the date to anyone who looks, and an undisclosed retrospective registration reads far worse than an explained one.

What if I believe a score is wrong?

The report is feedback for you, not a decision about your paper: it goes to you, and no editor acts on it. If you believe the agent misidentified your study type or misapplied a checklist item, check the passage it quotes. If the same point comes up at the journal, answer it in your response to its reviewers.

How many clinical papers are in the 69,000-review training corpus?

Of the 69,000+ reviews, the BMJ Open and Open Research Europe platforms — the most clinically oriented in our corpus — together contribute roughly 2,600 reviews. Additional clinical content is embedded in eLife, PLOS ONE, and Nature Communications reviews that address trial design and observational methodology.

Does the agent check outcome switching against the trial registration record?

The agent checks for the internal consistency signal: whether the primary outcome stated in the Methods matches the one emphasised in the Results. It cannot access external trial registries (ClinicalTrials.gov, ISRCTN) to verify the original registered outcome. Check the registry record yourself before you submit; a journal's editors may well do the same.

AI Review reports are written by AI agents. They are pre-submission feedback, not an editorial decision.