Dataset card
Science AI Journal Calibration Corpus
5,000+ real peer-review texts from 15 open-review platforms: the index 7 of the AI Review agents draw their calibration examples from. Our aggregate summary data is CC BY 4.0; the review texts stay under their platforms’ terms.
Sources
The corpus is aggregated from public open-review pages across 15 scholarly platforms. We count distinct texts: each referee report, reviewer follow-up, editor’s decision or assessment, public comment or authors’ response once, however many times the index stores it. Measured on 2026-10-28; where a platform mixes review text with other material the count is estimated from a hand-read sample, elsewhere every text was read or classified. In all, about 5,598 texts (conservative floor 5,092), quoted as 5,000+.
Not counted here: besides these texts, the retrieval index the agents read holds other text stored with them (article and front-matter text from journal pages and from papers in our library, which we do not count as reviews), sample reviews written as examples rather than collected, and three reviewer reports on a paper of our own. None of it is counted, part of this dataset or offered under its licence. The agents’ examples are drawn only from the 15 counted platforms, so the article text and our own reports never reach an agent. We also collect from PLOS ONE, BMJ Open, ScienceOpen and Royal Society Open Science, but the text stored from them is article text, metadata or rating-only stubs, so they add no review texts.
| Platform | Peer-review texts | Focus |
|---|---|---|
| OpenReview | 3,033 | ML, CS, AI conferences (NeurIPS, ICLR, ICML and more) |
| SciPost | 818 | Physics, CS, mathematics — referee reports |
| PREreview | 473 | Preprint reviews, posted openly |
| PeerJ | 302 | Life, environmental and computer sciences |
| Copernicus journals | 217 | Earth & environmental sciences (open discussion) |
| Journal of Open Source Software | 209 | Research software — open review on GitHub |
| Open Research Europe | 181 | Multidisciplinary (EU-funded research) |
| Peer Community In | 134 | Recommendation-based review of preprints |
| Nature Communications | 115 | Multidisciplinary (transparent peer review files) |
| eLife | 82 | Life sciences — decision letters and assessments |
| Review Commons | 12 | Journal-independent review of preprints |
| PubPeer | 10 | Post-publication comments |
| F1000Research | 4 | Life sciences — open post-publication review |
| Qeios | 4 | Multidisciplinary open review |
| EMBO Press | 4 | Life sciences — review process files |
| Total | 5,598 | Conservative floor 5,092 |
How the corpus maps onto the 9 agents
Texts are stored with their review-form section (weaknesses, strengths, summary, questions or the full report) and indexed with SQLite FTS5. At review time, each of the 7 calibrated agents searches that index with its own review vocabulary and takes up to 60 real examples from open peer review as Retrieval-Augmented-Generation context — the same examples whatever the manuscript (the Research Gap agent reads comparable published papers instead, and the Prior Publication agent is a deterministic lookup; neither uses this corpus). The aim is that a "methodology" agent reviews against how referees write about methods, not as a generalist.
Field mix
Each text is labelled with its platform’s field, not the paper’s discipline, so the mix is measured by where the texts come from: about 54% machine learning and computer science (OpenReview), 14% physics (SciPost), 13% life, health and earth sciences (PeerJ, Copernicus journals, Peer Community In and others), 8% preprint reviews and comments across fields (PREreview and PubPeer), 5% multidisciplinary journals (Open Research Europe, Nature Communications and Qeios) and 3% research software (Journal of Open Source Software). The examples an agent is shown are narrower still: on 28 October 2026 they came only from OpenReview, SciPost, PeerJ and JOSS.
Calibration method
Each of the 7 calibrated agents is prompted with up to 60 real examples from open peer review, drawn from this corpus and chosen for its own concern rather than the manuscript’s field, so it reasons against how referees write about that concern rather than against a generic prior. On 28 October 2026 every agent’s examples came only from four of the platforms: OpenReview, SciPost, PeerJ and JOSS. Each text may carry an outcome label — accept, revise or reject — which is the editorial decision where the platform publishes one; on several platforms (PREreview, PeerJ, Copernicus and the PCI communities among them) our scrapers infer it from the review’s own wording or set a default. Counted in distinct texts on 2026-10-28, 86% carry such a label — 58% accept, 18% reject, 10% revise; the rest are unlabelled. We have not published a measured agreement rate between our agents and human editorial decisions, and we are not going to quote one we have not run. The numbers we have actually measured, with their method, are on our benchmarks page.
A weekly calibration report summarises the stored corpus by agent tag — score-range coverage, label balance, platform diversity — and flags gaps for a person to act on; it does not replay the examples each agent is shown, and nothing is adjusted automatically. Reach out if you want the raw numbers for a research paper.
Frequently asked questions
- Is the dataset downloadable?
- A summary aggregate is available at /api/dataset/summary as JSON; that summary data is ours and is CC BY 4.0. The review texts themselves come from public open-review pages and stay under their source platforms' terms: we share parsed JSONL tranches on request with academic collaborators, under each platform's licence.
- How is the data used?
- Exclusively for Retrieval-Augmented-Generation (RAG) calibration of 7 of the AI Review agents. Each of those agents' prompt is prefixed with up to 60 real examples from open peer review that match its own concern, not the manuscript's field — weighted towards higher-quality texts and a mix of accept, revise and reject labels, with no platform above 40% of an agent's examples unless the others run out — to show the agent how referees write about that concern. The Research Gap agent reads comparable published papers instead, and the Prior Publication agent is a deterministic lookup.
- Do you re-host reviewer comments that were posted under pseudonyms?
- No. This site publishes the corpus as aggregate counts (the summary JSON); review text itself is not served. The calibration index keeps each text as its platform published it, with the source platform recorded, and excerpts of it are placed in the agents' prompts at review time. Licence terms differ from platform to platform: OpenReview review text is CC BY and eLife's published reviews are CC BY; any research tranche follows each source's terms.
- Does the dataset include the papers themselves?
- No — the dataset is the review texts, not the manuscripts. Each text is stored with the title of the paper it discusses and, where the platform shows one, its abstract; the papers themselves are not included.
- How do you keep calibration fresh?
- Scrapers re-run on rolling schedules. A weekly calibration report (scripts/training/calibrate_agents.py, Fridays) summarises the stored corpus by agent tag — score-range coverage, whether the score-to-decision mapping is consistent with the labels, platform diversity, and section-type balance — and flags gaps for a human to act on. It reads the stored rows, not the examples each agent is actually shown (the agents' search does not use the tag); it makes no model calls and does not adjust anything automatically.
- Can my institution contribute a tranche?
- Yes — if your journal or conference has transparent review records you want included, email [email protected]. We credit source and preserve licence terms.