Inside the Methodology Agent: How AI Catches Sample-Size Errors

The methodology agent is one of eight independent reviewers in Science AI Journal's review engine. It reads the full manuscript — up to 100,000 characters for Claude-backed runs — and applies a rubric with venue-specific criteria for machine learning, biology, clinical, physics, earth-science and ecology papers. This post opens that rubric and explains how the agent catches the sample-size and design errors that slip past busy human reviewers.

Why sample size is the hardest thing to catch in peer review

Underpowered studies are the most common serious flaw in published research. Button et al. (2013) found that the median statistical power across a sample of neuroscience studies was 21%, meaning most studies were not large enough to reliably detect the effects they claimed to find (doi:10.1038/nrn3475). The problem has since spread to machine learning, clinical AI, and any field where benchmark experiments substitute for adequately powered studies.

Human reviewers miss these errors for understandable reasons. Calculating post-hoc power for a paper in an unfamiliar sub-field takes time and statistical facility that unpaid, overloaded volunteers rarely have. A quick scan of "n = 45" looks reasonable without working through what effect size and alpha level that sample can actually support. The methodology agent has no such constraint. It applies the calculation to every paper, every time, using threshold values derived from the venues in the 69,000-review training corpus.

What the agent actually checks

The methodology agent's evaluation framework is defined in journal/src/components/hakem/ScienceAI.tsx. It branches by publication context, applying different criteria to machine learning papers, clinical trials, biological studies, and physics or mathematics work.

For machine learning papers (calibrated against NeurIPS, ICLR, and ICML via the OpenReview corpus), the agent checks:

• Iso-compute and iso-parameter baselines. Comparing models with different FLOP budgets or parameter counts without disclosing the difference is a systematic rejection trigger at NeurIPS and ICLR. The agent flags any comparison where architecture or compute scale is not matched or explicitly controlled.
• Seed coverage. Single-run results without seed averaging are insufficient for ICLR and ICML. The rubric requires a minimum of three seeds and five or more for high-variance tasks such as reinforcement learning.
• Ablation completeness. A claimed contribution without an ablation study has not demonstrated that the contribution is the causal factor in any observed improvement.

For biology and life sciences papers (calibrated against eLife), the agent checks:

• Whether n values refer to biological replicates or technical replicates. Three technical replicates, repeating the same measurement on the same sample, is not the same as three biological replicates, and the agent flags the conflation explicitly.
• Whether blinding during data acquisition and scoring is stated. eLife requires an explicit statement; absence is flagged.
• Whether sample size was pre-specified or chosen post-hoc. "Sample size was what was available" without a power justification is a red flag in the rubric.

For clinical research (calibrated against BMJ Open and the CONSORT 2010 checklist), the agent examines:

• Whether a power calculation appears in the Methods section with the target effect size and alpha level stated.
• Whether the primary outcome is clearly pre-specified. Any paper that appears to have switched its primary outcome after data collection is flagged as a critical concern.
• Loss to follow-up: a drop rate above 20% triggers a selection-bias flag under the STROBE framework for observational studies.

How the agent generates specific, actionable feedback

The rubric does not produce vague warnings. It produces numbers. The worked example embedded in the evaluation framework, and visible in the agent's typical output for a clinical paper, reads:

"Sample size of n = 45 per arm provides only 62% power to detect Cohen's d = 0.5 at alpha = 0.05; minimum n = 64 per arm required for 80% power."

This level of specificity is difficult to sustain across a full review load when the reviewer is calculating manually across dozens of submissions per month. The agent provides it for every submission.

The framework also captures the structural version of underpowering: pseudoreplication. When technical replicates are entered into a statistical model as independent observations, the effective sample size is inflated and significance thresholds are artificially easy to cross. The rubric names it directly: "Pseudoreplication: treating technical replicates as independent observations in the statistical model."

Venue-specific standards built into one rubric

One feature of calibrating on real peer reviews rather than constructing a generic rubric from scratch is that venue-specific expectations survive intact. A few examples of how the framework diverges across fields:

• SciPost and physics papers: "We show X" requires a proof. "We conjecture X" requires a clearly labeled hypothesis. The conflation is flagged. Proof-by-example fallacy, demonstrating a property holds for three cases without proving it holds generally, is an explicit catch.
• PCI ecology papers: Occupancy models, N-mixture models, and mark-recapture analyses must account for imperfect detection. Omitting detectability is a primary-round rejection trigger in PCI Ecology, and the agent applies that standard.
• Royal Society Open Science: Data accessibility statements must include a repository DOI or accession number. The rubric flags any paper whose data statement says "data available on request" instead.

This specificity comes from calibration on the 69,000-review corpus, drawn from OpenReview, eLife, BMJ Open, Nature Communications, SciPost, PLOS ONE, F1000Research, Copernicus, PREreview, Qeios, Royal Society Open Science, PCI communities, and ScienceOpen.

HARKing: the problem that looks like hypothesis-driven science

Beyond sample size, the methodology agent flags HARKing, or Hypothesizing After Results are Known: papers whose introductions present post-hoc findings as if they were a priori hypotheses. This is structurally hard for a human reviewer to detect without access to preregistrations, because the narrative is designed to read as planned science.

The agent's approach: any mismatch between "we hypothesized" language in the introduction and the absence of a preregistration link is raised as a question for the authors. The rubric treats this as a flag, not an automatic rejection, because some legitimate retrospective studies use hypothesis framing appropriately. The flag prompts an author response rather than a silent score penalty.

How scores translate to real decisions

The methodology agent scores on the same 1-10 scale as the other seven agents in the eight-agent review engine. The bands are calibrated against acceptance thresholds at the venues in the corpus:

A score of 5 on methodology reflects the same class of problem that produced major-revision recommendations at eLife or BMJ Open in the training corpus. A score of 3 reflects near-reject language from the same platforms. The scale is calibrated, not invented. For a full walkthrough of how calibration works across all eight agents, see how we calibrated on 69,000 real peer reviews.

What this means for authors

Before running a paper through AI Review, the methodology section is where preparation returns the most value. Concretely:

• Report a power calculation in Methods. If the study is retrospective, acknowledge that the sample size was determined by data availability and what that implies for power.
• Distinguish biological from technical replicates in the study design description. A single sentence clarifying this removes one of the most common major-revision flags in biology papers.
• For ML papers, report results across at least three random seeds and use the same compute budget for all compared models.
• If the primary outcome changed after data collection, state this. The rubric penalizes concealment and rewards transparency in the "questions for authors" output.

Our research-gap finder searches the papers in your field before you write, and a topic unlock, paid in credits (see pricing), writes the research gaps from them. That can help calibrate how thorough your Methods section needs to be relative to the existing literature.

For more on where the review engine's limits are, including what the methodology agent cannot reliably catch, see when AI peer review falls short.

Frequently asked questions

Does the methodology agent replace a human statistician? No. It applies a rules-based rubric calibrated on real reviewer decisions from the platforms in its calibration corpus. A human statistician reviewing a novel causal inference design brings domain reasoning the agent cannot replicate. The agent reliably catches common, well-defined errors, including underpowering, pseudoreplication, and missing controls, faster and more consistently than an overloaded volunteer reviewer.

What if my paper is in a field not explicitly covered by the rubric? The agent applies the closest applicable framework. For interdisciplinary papers, it uses the general red-flag list, which covers circular reasoning, undisclosed outlier removal, and causation claimed from correlation. These apply across fields regardless of publication venue.

Why does the methodology agent sometimes give a lower score than I expect? The rubric is calibrated against acceptance thresholds at high-quality open-access venues. A score of 6 reflects the standard of a real major-revision decision at eLife or BMJ Open, not a judgment about the importance of the research question. A well-designed study with a clear sample-size justification and appropriate statistical test typically scores in the 8-9 range.

Can I see the rubric? Not in full: the rubric is part of the review engine's source code, which is not public. This post quotes its main criteria, score bands and worked example.

Is the rubric the same for a physics paper and a clinical trial? No. The rubric branches explicitly by field. ML papers are checked against OpenReview standards, clinical papers against CONSORT 2010 and STROBE, ecology papers against PCI Ecology's detection-probability requirements, and physics papers against SciPost's proof-completeness standards. The score bands are common; the criteria for reaching each band differ.

AI Review reports are written by AI agents. They are pre-submission feedback, not an editorial decision.