Verifying the Verifiers: Towards Autonomous Policy Evaluation

AI is now capable of producing policy evaluation papers, end to end, that look like real papers. But are they to be trusted? If they can be, the technology would open the door to evidence generation at unprecedented scale: cheap, rapid, and global. Checking thousands of papers by hand is infeasible, so verification itself must be automated — which raises the question of whether the verifiers can be trusted. Here, we study the reliability of automated verifiers powered by LLMs. We first develop a versioned taxonomy, the CRED codebook (Classification of Research Errors and Defects), followed by a benchmark and scoring method that quantify a verifier's ability to detect errors under that taxonomy. This lets us compare verifiers and trace the frontier between cost and detection recall over time. We find remarkable recent progress. Eighteen months ago, with the first ‘reasoning’ models, detection recall was around 30% — more than two of every three errors missed. By February 2026, the best models missed one in six. In July 2026, the first model reached 99% recall (Codex 5.6 Sol), suggesting a new era for verifiers.¹1. The 99% figure is Codex 5.6 Sol's detection recall on the single-error pool. Precision uses Codex 5.5 high, which was the recall leader when selected. The two-error and replicate panels cover selected configurations but do not yet include every newest model.²2. Cf. Andrej Karpathy's “march of nines”: every additional nine of reliability takes roughly as much work as all the nines before it — a framing from self-driving that we discuss for research automation on our autonomy page. Costs have collapsed alongside: the cheapest price of 50% recall fell roughly 90 times over the past year, and the cheapest price of 20% recall fell more than 400 times in eight months. Ensembles built largely from models with open weights prove highly cost effective. Reliability, however, is not solved: recall degrades when a paper contains multiple errors, and confidence scores remain too poorly calibrated to decide when human review is unnecessary. We conclude by discussing what is verifiable in principle and what is not. Together, our results suggest that automated verification of policy evaluation research may soon be feasible at scale, reliably and cheaply.

The cost of producing social science research appears to be collapsing. Core parts of the research generation process — ideation, data collection, analysis, and writing — are increasingly being powered using AI.³3. Lu et al. (2024), The AI Scientist; Schmidgall et al. (2025), Agent Laboratory; Si, Yang & Hashimoto (2024) on LLM-generated research ideas. The reduction in cost could, in principle, open the door to standardized policy evaluation at scale, given the vast unexplored space of policies around the world combined with the ever-growing availability of high-quality data. Done at scale, evidence-based policy making could therefore stand on much more solid ground.

However, if AI-generated research is unreliable, filled with errors, then production at scale is a bad idea. Hence, verification becomes key. Here, just as on the generation side, while the cost of verifying output may also be falling, whether we can reliably leverage AI to act as an automated verifier remains an open question. In short, the frontier between cost and detection recall for automated verifiers is unknown. We attempt to identify it.

Our setting is the Autonomous Policy Evaluation (APE) project, where from January to April 2026 we generated 1,000 empirical economics papers on policy evaluation from across the globe.⁴4. Policy evaluation typically concerns a policy such as a tax, subsidy, law, or regulation and its causal effect on one or more outcomes. Estimating that effect requires observational or randomized data and an empirical strategy. Until now, the quality of these papers was unknown to us: reviewing them all was precluded by the cost of skilled labor. We will release our evaluation of these 1,000 papers soon. Any such assessment is only as good as the LLM producing it: the verifier must itself be verified. This is a version of the recursive monitoring problem studied by Rahman (2012): delegating potentially costly verification to a monitor shifts the question to how the monitor itself can be assessed. Moreover, given the plethora of possible models and harnesses that could be leveraged, in principle many verifiers could be used.

We pursue the ideal that anything that can be verified, should be verified. In practice, it is not obvious what that entails. For example, imagine that an AI-generated paper claims that it studies some policy where it downloaded panel data from the U.S. Census Bureau, used a two-way fixed effects regression approach, and estimated a statistically significant positive effect of 15 percentage points with a p-value of 0.02. Can an automated verifier truly scrutinize these claims? A good verifier should. For each, there is a ground truth, and if there is an error, a reliable verifier should always detect and report it.

To make progress, we develop a versioned taxonomy for classifying errors in policy evaluation research. We call it the CRED codebook, short for ‘Classification of Research Errors and Defects’. The codebook is developed in order to detect critical types of errors that a policy evaluation paper could have. We do not claim that CRED covers all possible verifiable errors and defects that could arise in APE. However, we do believe that it covers necessary, first-order concerns required by a verifier. In fact, much like in other domains where taxonomies evolve over time, we expect CRED to evolve. (For example, in human disease classification, the International Classification of Diseases is on its tenth revision, ICD-10.) Here, we report results using our current version, CRED v0.92. This also means that we only report benchmark results on a verifier's ability to detect errors, given an error classification scheme. Armed with CRED, we concretely ask three questions:

Below, we report results after describing in detail how we overcome conceptual and practical challenges. In short, challenges fall into three broad domains: 1) establishing a classification scheme and measuring the ground truth, 2) developing deterministic components of verifiers that minimize the stochastic properties of LLMs, 3) searching for the most cost-effective verifier. To our knowledge, CRED is the first classification scheme for policy evaluation research. It is tailored to help solve a key bottleneck for Project APE, where the north star is end-to-end autonomous policy evaluation. We are also unaware of any previous work identifying the frontier between cost and detection recall for automated verifiers in empirical economics.

Conceptually, our framework relates to Rahman (2012), who studies how to assess a monitor when monitoring is costly and its observations are private. His mechanism uses occasional hidden tests for which the principal knows the correct answer. CRED applies the same measurement logic: we inject errors whose locations and classifications we know, keep the instances and answer key from the verifier, and score its findings against that ground truth.

While a growing literature studies the application of AI to the generation of scientific research, work evaluating LLMs as verifiers of research is just starting. SPOT pairs 83 published papers with 91 known errors. Each error is severe enough to warrant an erratum or retraction, they claim. In their paper, the best 2025 model reaches 21.1% recall and 6.1% precision, with low confidence and high run-to-run variance. We believe these results more likely reflect model capability at the time than inherent task difficulty, especially given the results we report below about model capabilities over time.⁵5. Son et al. (2025), SPOT. Adjacent evidence points the same way: automatic reviewers largely fail to detect faulty reasoning under counterfactual evaluation (Dycke & Gurevych 2025), while LLM analysis surfaces errors at scale in published AI papers (Bianchi et al. 2025).

FLAWS injects claim-invalidating errors into accepted computer-science papers and scores a ranked list of candidates; the best model puts the injected error at the top of its list 9% of the time. We do not adopt a ranked metric: our host papers are not guaranteed to be free of other defects, so a detector may legitimately surface a more salient real error than the injected one, and a rank-based score would penalize exactly that behavior.⁶6. Xi et al. (2025), FLAWS: accuracy@k of 9.0%, 19.2%, and 39.1% at k=1, 3, and 10 for the best model (GPT-5).

These studies task LLMs with identifying textual errors in PDFs. Empirical economics today, however, involves collecting and processing large datasets across numerous code files: errors can occur in the manuscript, in the code, or between the two. SciCoQA studies exactly this on human-written computational-science papers; the best models reach roughly 40% recall on real errors. We differ on three axes: we inject errors into self-reproducing, AI-generated host papers, providing exact ground truth; we score with a deterministic matcher rather than an LLM judge; and we decompose reliability by varying the number of injected errors. To our knowledge, this is the first benchmark of frontier LLMs on identifying and localizing errors specific to empirical economics.⁷7. Baumgärtner & Gurevych (2026), SciCoQA: 635 paper–code inconsistencies. Reproduction benchmarks (CORE-Bench, Siegel et al. 2024; REPRO-Bench, Hu et al. 2025; Kohler et al. 2026) score regenerated results rather than detected errors — a complementary task.

This section outlines how we use the CRED benchmark to solve our first two challenges: developing a taxonomy of errors for which we can measure ground truth, and creating a deterministic scoring mechanism that eliminates variability from scoring with an LLM-judge. We achieve this by injecting known CRED errors into papers that demonstrably reproduce, and by scoring each verifier with a deterministic text match rather than an LLM judge.

We distinguish four components throughout. The CRED codebook is the versioned taxonomy of error classes. The CRED benchmark suite is the collection of host papers and injected errors used to test detectors.⁸8. To preserve CRED as a valid benchmark for future models, we do not publicly release the mutated benchmark instances, their answer key, or raw detector outputs that reveal benchmark content. Public release could allow these materials to enter training data or be used for targeted tuning. A model could then learn answers specific to the benchmark without acquiring the underlying ability to verify unfamiliar papers (Jacovi et al. 2023). This mirrors Rahman's trick question logic: the test is informative only while its answer remains hidden from the monitor. We will, however, release the detector prompt template, output schema, scoring and analysis code, configuration metadata, and derived data needed to reproduce the reported analyses and figures shortly after publication. The scoring protocol deterministically matches detector findings to those injected errors. A detector configuration is the particular model, prompt, reasoning-effort setting, and delivery system being evaluated.

The Replication and Build Defects (RBD) gate is the deterministic foundation of our benchmark: it establishes whether a replication package reproduces before we use its paper as a host into which we inject errors. The RBD pipeline first runs the package's R code end to end and records whether execution succeeds. For packages that pass this step, it then compares the regenerated tables with the tables in the paper, allowing for rounding. Together, these results form the reproduction record. We run RBD first to select host papers that reproduce, and again on every error-injected copy used as a benchmark instance. This stage is purely deterministic: it runs the supplied code and records its output. The LLM detectors receive this reproduction record; they do not execute the code themselves. A failure to execute is therefore directly observable in the record, whereas other errors require comparing the manuscript, tables, and code.

The papers used as hosts are drawn from the APE corpus of 1,000 papers. A paper passes RBD when its code executes successfully and its regenerated tables match the tables in the paper. The gate yields 144 eligible papers. The design is mutation testing applied to policy evaluation: we insert known errors into papers that demonstrably reproduce and measure each detector's ability to identify them.⁹9. Mutation testing: DeMillo, Lipton & Sayward (1978), “Hints on Test Data Selection,” IEEE Computer; Jia & Harman (2011) survey, IEEE Trans. Software Engineering. Because the host self-reproduces before mutation, the injected edits are the only known discrepancies.¹⁰10. Hosts may still contain latent defects we did not inject, but these cannot inflate recall: a finding earns credit only if it quotes the injected edit itself.

The 100 hosts are not a random sample, and we make no claim to represent a population. Passing RBD is the sole benchmark eligibility criterion. From the 144 papers that pass, we selected 100 for feasibility and the planned chapter mix, as we aim to identify additional errors.¹¹11. On design-based rather than sampling-based inference: Abadie, Athey, Imbens & Wooldridge (2020), Econometrica. On benchmarks as instruments, not surveys: Raji et al. (2021), NeurIPS Datasets and Benchmarks. Each injected error comes from our CRED taxonomy, whose four chapters classify four types of observable defects:¹²12. The codebook measures reproducibility in papers, not correctness (National Academies 2019; Goodman, Fanelli & Ioannidis 2016, Science Translational Medicine); codes describe what is visible in the paper and its code, so no author intent is inferred.

• Prose–Table Inconsistency (PTI). A value or claim in the manuscript disagrees with the corresponding cell in the tables. This can be, for example, a numeric mismatch, a sign flip, a significance flip, or a unit disagreement.
• Paper–Code Inconsistency (PCI). The manuscript disagrees with what the code does. Examples: a different estimation choice, an inaccurately stated sample, treatment timing, or variable construction.
• Execution/Reproducibility (EXE). The package does not self-reproduce: the code fails to run end to end, or a rendered table cannot be regenerated from the current code.
• Data Integrity (DAT). The inputs are synthetic, fabricated, or transformed using outcome values. For example, the code may generate synthetic data instead of using real data.

We designed and injected each error using the codebook and examples from the corpus as references. We reviewed every mutation individually before applying it to a host. Because we control both the original package and the exact injected edit, we know the error ground truth and location exactly.¹³13. The mutated segments were created in June 2026, after the training-data cutoff of the models in the initial benchmark suite; more fundamentally, ground truth is agreement within the package — so a memorized unmutated package has no effect on the outcome. Cf. Kohler et al. (2026, ETH Zurich), who find no pre- versus post-cutoff performance difference in the adjacent reproduction setting. From the same 100 hosts we construct three arms: arms A and B each insert one error per host (pooled as the single-error pool, n=200), and arm AB inserts both errors at once. Recall can then be compared on the same error with and without a second error present. Unless a figure says otherwise, results use the single-error pool (n=200), so every model is scored on the same 200 instances under the same protocol.

During benchmark construction, we also assigned each error a coarse severity annotation based on its likely consequence for the paper: severe errors could invalidate a headline result, such as through corrupted input data; moderate errors are materially wrong but contained; and minor errors are localized issues, such as a rounding inconsistency. These are reviewed benchmark annotations, not independently validated severity ground truth, so comparisons by severity are descriptive.

The detectors

We define a “configuration” as a combination of an LLM, a prompt, and a reasoning-effort setting. Configurations come in two types: single-shot configurations, which receive the replication package combined with the prompt in a single API call, and agentic configurations, which receive the same codebook and task instructions but are provided the replication folder as a workspace, with read-only tools. For comparability across these two types, every configuration uses the same codebook, task instructions, and output specification. Agentic harnesses enforce those fields with a JSON Schema; single-shot responses are parsed against the same fields. Reasoning-effort labels are specific to the models and ordinal only within a model family; for example, a “high” setting is not necessarily assumed to be comparable across families. After reading the replication package for a paper, every configuration returns a structured list of findings: verbatim quote, location, chapter, one-sentence rationale, proposed severity, and confidence (0–100%). While we do not claim to have tested every available model, we generally selected models that were viewed as near the frontier on common benchmarks at the time or that offered a promising price point. We select configurations to span three axes that plausibly shape verification cost and capability: token price, which ranges over roughly two orders of magnitude; open- versus closed-weight release; and developer geography.¹⁴14. A configuration also includes the delivery harness, where applicable (for example, Claude Code), and the harness version used for the run. The same model run through a different harness is a different configuration.

To compare configurations on a common metric, we reconstruct an equivalent API cost per paper from recorded token counts using public prices as of July 14, 2026. For single-shot configurations, this closely corresponds to the metered API charge from OpenRouter. Agentic configurations were run through subscriptions, so their plotted cost is an API-equivalent price. Plausible price changes do not change the ordering of the frontier.

Three analysis samples. The analyses use one “capability” sample and two “analysis” subsets. The capability sample contains every configuration with completed single-error arms A and B and currently includes 64 configurations. The interference panel contains the 18 configurations that also completed the two-error AB arm. The repeatability panel contains the 10 configurations with an independent rerun on the same 100 A instances. A configuration enters each sample only after completing the runs required for that analysis. Finally, failed calls and refusals count as misses, and the sample used by each figure is stated in its subtitle.

Scoring

We choose to score the findings without using an LLM to avoid potential measurement error. Rahman describes checks for which the principal knows the correct answer as “trick questions”: because the answer is known, the monitor can be scored directly. Injected CRED errors serve the same function. Using another LLM as the judge would move the monitoring problem back one step because the judge would itself need to be evaluated. Its verdict can change with the prompt, judge model, temperature, or answer order. LLM judges can also favor verbose or self-generated answers, and their agreement with human judgments varies substantially across tasks.¹⁵15. LLM judge outputs vary with model and measurement choices (Messing 2026) and answer order (Wang et al. 2023). They carry verbosity bias (Zheng et al. 2023, NeurIPS) and self-preference (Panickssery, Bowman & Feng 2024, NeurIPS), while their agreement with human experts varies across tasks (Bavaresco et al. 2024). Deterministic matching instead makes the score reproducible: given the same finding and injected error, it returns the same result.

We credit findings using deterministic string matching, and the setup is as follows. Every detector provides its findings in a fixed format, containing a verbatim quote (the evidence), a location, a rationale (which is not used for scoring), and a chapter label. A finding ties to the injected error if its quoted evidence contains the changed segment itself (plus a few characters of context); quoting surrounding text is not sufficient. In the two-error (AB) arm, findings are matched one-to-one to errors (each finding can be credited against at most one error, and vice versa), so a single quote cannot be credited twice. A quote can be credited in three ways (Figure 3):

• Window containment: the quote contains the changed segment plus a few (4–12) characters of context. Containment is verified twice: on the raw text, and after canonicalization (LaTeX stripped, math symbols and typography normalized, whitespace removed).
• Long-edit fragment: for long inserted or replaced segments, which detectors quote in pieces, the quote must share a contiguous substring of at least 18 characters with the modified segment itself.
• Short-edit full containment: for edits too short to form a distinctive window, the quote must contain the entire edited text, original or mutated, verbatim.

These character cutoffs were selected while developing the matching logic, after auditing cases in which the matcher either credited the wrong finding or failed to credit a genuine detection. The matcher first requires the changed text together with 12 characters of context on each side, which makes accidental matches unlikely. It loosens this requirement to four characters when a detector quotes only the changed value and its immediate context. For long edits, an 18-character overlap with the changed text allows truncated quotations while still requiring a substantial verbatim match. We fixed these rules and applied them uniformly to every error and detector configuration.

On papers where the injected error is EXE, a finding labeled EXE that quotes the RBD result is credited, since an execution failure shows up in the execution run result, and there is no edited line to quote. This is the only crediting rule that uses the finding's chapter label; requiring EXE here prevents unrelated references to RBD from matching. These rules can under-credit correct paraphrases, so we interpret reported recall conservatively.¹⁶16. Concretely: of the strongest run's two uncredited errors (Codex 5.6 Sol, 99%), one was in fact found and quoted verbatim — but the quote ended two characters after the changed digit, short of the matcher's context window, so it earned no credit. The full specification, with window sizes, thresholds, and a worked example, is in Appendix D.

The scoring measures two tasks:

• Detection (localization): an injected error counts as detected if a finding ties to it.
• Attribution (chapter-correct): an injected error additionally counts as attributed if the tying finding's chapter label equals the error's true chapter.

Attribution is therefore a strict subset of detection, highlighting the gap between finding an error and correctly assigning it to a CRED chapter. An example of this is as follows: the detector quotes the exact line where the paper contradicts its own code (a PCI error), but classifies it as a prose–table inconsistency (PTI), which is detection without attribution (the dominant pattern in the data; section 4). We report conditional attribution, which is the share of correctly labeled errors, conditional on detection.

At the end of 2024, the best available model (o1) caught 30% of errors. By February 2026 the best model missed one in six, and in July 2026 we recorded the first model reaching 99% recall.

We measure recall by tasking the 64 configurations with identifying CRED errors in each of the 200 mutated copies of our 100 host papers. Single-shot and agentic configurations are tested slightly differently. Single-shot models are run using OpenRouter, a gateway to many labs' models, which charges per token. The same model is sometimes served by multiple providers at different prices, which is why we reconstruct costs based on the recorded token counts at each model's public list rate. Agentic configurations are run on our machines through the Claude Code or Codex Command Line Interfaces (CLI) using subscriptions that don't charge per token. Agentic results measure the model together with the harness version used at run time, and harnesses update over time, so an older model run through a later harness may do better than it would have at release. Every model was scored between June and July 2026. A model released in 2024 was therefore measured in 2026: its position on the release-date axis says when it became available, while the measurement itself is current.

The evolution of recall over time is shown in Figure 4. Models released by July 2024 cluster around 12–14%, but by January 2025, recall climbs to 30–40% with the introduction of the first two reasoning models, o1 and DeepSeek R1. Recall then continues rising, reaching 80% in February 2026, and less than six months later, 99% recall is achieved by Codex 5.6 Sol (high).

The set of configurations at the frontier of cost-recall has also evolved over time. For each dated group, we plot the best available cost-recall trade-off by that date, keeping only undominated configurations (dominated grey points are models beaten on both price and recall). Since 2024, a time when the frontier was both weak and expensive, the frontier moved up and left, as recall increased and cost decreased for the same fixed recall.

Instead of measuring what the best model can achieve, what happens when we keep the recall level constant and ask what the cheapest way to reach it is? In Figure 6, we plot how a fixed amount of recall has changed in price over time. The cheapest price for 50% recall has fallen roughly 90× per year over the past year, while 20% recall fell over 400× in eight months. This measurement highlights how increasingly high recall is becoming dramatically cheaper.

We find that recall is not uniform across different types of errors. Detection averages 99% on execution errors, but this is by design: every detector receives the reproduction diagnosis (the log of whether the code ran and the tables reproduced) as input, and the execution result is written in it. To detect an execution error, a model only has to read it. Data integrity stands at 75% and prose–table at 58%. Paper–code inconsistency, at 43%, is the hardest chapter: a paper–code contradiction is written nowhere in the package — it exists only between the manuscript and the code, and must be constructed by reading both.

The average PCI error is caught by 43% of the configurations, and the residual misses of the strongest configurations are concentrated in the consistency chapters — both errors the 99% run left uncredited are consistency errors.

The same pattern holds error by error. We can group the injected errors by what we know about them: chapter, severity annotation assigned during benchmark construction, and which file the edit modified. We can then ask which kinds of error the detectors miss most often. For example, an error planted in the analysis code is caught by 77% of configurations on average, while an error planted in the manuscript is caught by 49%.

We also study whether our detector assigns errors to the right CRED chapter. For example, if it really is a prose–table inconsistency (PTI) and not a paper–code inconsistency (PCI), does the detector assign it correctly? Our results show that it does a good job of this: among credited findings, it names the correct chapter 92% of the time (pooled across configurations). To see this, consider the confusion matrix below. Here, we compare the detector's attribution against our ground truth. Most of the misattribution sits in a single cell of the confusion matrix: 19% of credited PCI findings are emitted as PTI, which accounts for 81% of all misattribution in the pool. We speculate that this misattribution likely occurs because the detector has to perform an extra step. That is, it may quote the right line but name the wrong mechanism, recording the discrepancy without registering that establishing it required reading the code.

Increasing test-time compute, often called “reasoning effort,” is a well-known way to improve the capabilities of language models. We first show how recall improves for a set of commonly used closed-weight models as we increase compute within each model.

Within a model family, the reasoning-effort setting acts as a price dial, as higher reasoning typically equates higher token usage. Codex 5.5 climbs from 88.5% to 96.5% recall across its effort ladder for 1.9 times the cost, Sonnet 4.6 climbs from 72% to 83.5% for 2.6 times the cost, and Codex 5.6 Sol climbs from 94.5% to 99% for 2.0 times the cost.

So-called “ensembles,” which orchestrate outputs from different models, are another well-known way to improve performance.¹⁷17. Ensemble selection from libraries of models: Caruana et al. (2004), ICML; Dietterich (2000) on ensemble methods. One reason ensembles may be efficient is that different models introduce diversity: they miss different types of errors in papers.¹⁸18. Diversity, not redundancy, is what makes classifier ensembles work: Margineantu & Dietterich (1997), ICML; Kuncheva & Whitaker (2003), Machine Learning. Below, we investigate the cost-effectiveness of open-weight models as inputs to an ensemble search. We find that this approach can be quite cost-effective.

We combine configurations with an OR rule: an ensemble detects a defect if any member's credited finding does. We select members using a cost-aware variant of forward selection, adding at each step the configuration with the largest recall gain per additional dollar. To reduce tailoring to our particular 200 errors, we repeat the selection on 50 random subsets of the detector library, combine the selections by vote, and evaluate recall on a held-out 30% of the errors, averaged over five splits. Reported ensemble recall is therefore out of sample.¹⁹19. Within each split, errors from the same paper remain together.

With four members, the ensemble library reaches 94.7% recall at about $0.20 per paper. With 15 members, held-out recall reaches 100% at $0.83. The gain comes from diversity rather than redundancy: members miss different errors. The caveat is precision. An OR rule multiplies findings, and we do not measure the validity of the combined set of findings.

The diversity is visible directly. Contrary to our expectation, correlations in the errors that pairs of detectors miss show little clustering by lab. Configurations from the same lab do not consistently miss the same errors.

In the previous section, we show how LLMs are reaching near-perfect scores in error detection. These measurements were made in controlled settings. We only inject one error, run the detector once, and score against our own ground truth. Policy evaluation in practice is far messier, and papers contain an unknown number of errors. Automated verifiers may also struggle as errors compound and complexity increases, in the sense that the probability of catching one error could go down if there are other errors present. Since LLMs are stochastic, an automated verifier could report different errors across runs. Finally, a model's reported ‘confidence’ in error detection may be entirely spurious.

To assess robustness, we measure three more properties on the same instances: what happens when a paper contains a second error, whether per-error detection outcomes change between identical runs, and whether stated confidence predicts anything. Each one changes how the headline recall should be read.²¹21. We follow the framing of Rabanser et al. (2026), “Towards a Science of AI Agent Reliability”: capability and reliability are distinct axes, and reliability must be measured on its own. We show that reliability does not appear to order with capability or price: the current recall leader has AUROC 0.72, below the at-scale median of 0.74, on discrimination (how well its confidence separates findings that detect the injected error from those that do not) among detectors that emit findings at scale, and DeepSeek V4 Flash xhigh changes 15% of its per-error detection outcomes between two identical runs.

5.1 · Compounding errors

What happens to recall when we inject an additional error into the same paper? Adding a second error lowers per-defect recall for 17 of 18 configurations, with effects ranging from a 9.5-point drop to a 1.5-point gain (median drop of 4 points). Because the two-error arm reuses the identical errors, this suggests that any drop in recall is caused by the presence of the second error (the errors themselves are unchanged). The two-error arm covers the 18 configurations in the interference panel; the newest models have so far run the single-error arms only. Seven configuration-specific 95% intervals exclude zero.

The mechanism is visible in output volume: the number of findings a configuration emits per paper stays essentially flat when the injected defects double, so the same number of findings has to cover more errors. The drop concentrates in the consistency chapters (PCI and PTI), while DAT and EXE are stable under load, and its size is unrelated to single-error recall: configurations with similar recall differ widely in how much a second error hurts them. Robustness therefore has to be measured separately.²²22. The pattern echoes multi-target retrieval stress tests (Hsieh et al. 2024, RULER, COLM) and position effects (Liu et al. 2024, “Lost in the Middle,” TACL) — but appears here at ordinary context lengths.

This finding, that verifiers are less likely to detect a given error or defect if there are additional errors or defects, suggests that there is a potentially important interplay between automated generation of policy evaluation and automated verification of it. If the generator is worse, producing more errors, then the verifier will perform worse too. It is beyond the scope of the current research to study the functional form, but we will investigate this hypothesis in follow-up research in Project APE.

5.2 · Consistency: run-to-run variance

We run 10 configurations a second time — the subset with replicate runs, spanning the frontier — on the same 100 errors from arm A with unchanged settings; since scoring is deterministic, any disagreement measures variability across runs for the full detector configuration. Aggregate recall is highly stable: it differs by 0 to 4 percentage points between runs. Codex 5.5 high is nearly deterministic at the individual outcome level and changes 1% of outcomes. For mid-ranked models, individual outcomes are much less stable: up to 28% of errors are detected in one run and missed in the other. Because the flips roughly cancel out, aggregate recall looks stable even when many individual outcomes changed.²³23. Unlike repetition metrics such as pass@k (Yao et al. 2024, τ-bench), both runs are scored against known injected defects, so agreement is measured on ground truth rather than on self-consistency.

5.3 · Predictability: confidence calibration

We measure two aspects of stated confidence: calibration (does stated confidence match how often a finding detects the injected error?) and discrimination (does confidence rank findings that detect the injected error above those that do not?). Consider one comparison. Among findings above 90% confidence, 16% of Codex 5.6 Sol high's findings detect the injected error, compared with 84% of Opus 4.8's. The same confidence score therefore has different meanings across models. For every configuration that emits findings at scale, the share detecting the injected error falls short of stated confidence across the whole scale.²⁴24. Overconfidence in verbalized confidence is a general LLM finding: Kadavath et al. (2022); Tian et al. (2023, EMNLP); Xiong et al. (2024, ICLR).

Discrimination is heterogeneous. Among the 59 configurations that emit at least 150 findings, confidence ranks findings that detect the injected error above those that do not better than chance in 58 cases. AUROC ranges from 0.43 to 0.85, where AUROC is the probability that a finding detecting the injected error gets higher confidence than one that does not; 0.5 is chance and 1 is perfect. Confidence can therefore support triage for many configurations, but it is not universally informative and should not be read as a probability. Detection recall predicts neither calibration nor discrimination: the configuration with 99% recall has an AUROC of 0.72, while the best discriminator (Fable 5 med, 0.85) has detection recall of 96.0%. Among the configurations with list-order data, the detector's own list order serves the same purpose: 58 to 85% of credited findings appear first in the list. Calibration, discrimination, and list order move independently of recall and of each other, which is why we report them separately. None of them, however, answers the remaining question: whether the findings a detector reports are true. That is the subject of the next section.

The two previous sections measured what errors the verifiers missed. In this section, we investigate how many of their findings are real. To assess the verifiers' trustworthiness, recall alone is insufficient: a configuration can report every claim in a paper as an error and achieve perfect recall, all while being useless to the user. Rahman's motivating example makes the corresponding problem concrete: rewarding a monitor for reporting a mismatch can induce the monitor to report a mismatch without performing the costly check. Our precision exercise tests the analogous failure mode by examining whether the verifier's findings on unmutated papers are actually valid. Because we do not have a complete inventory of all defects in these unmutated papers, undetected defects cannot be observed. We therefore use the share of emitted findings that reviewers classify as valid defects as our measure of precision. Findings whose validity cannot be determined count as not valid, so the reported share is conservative.²⁵25. Rahman (2012) motivates the monitoring problem described here. The design (two reviewers, disagreements resolved by consensus) follows the STARD reporting standard for diagnostic accuracy studies (Bossuyt et al. 2015, BMJ). At scale, this metric matters because even a modest share that is not valid can create substantial human review work.

We run the configuration selected for the precision exercise, Codex 5.5 high, which was the recall leader when selected, on a separate sample of 100 unmutated papers. The sample is disjoint from the recall hosts. Using a fixed seed, we selected papers at random within method strata from 779 eligible papers in the APE corpus of 1,000 papers, with the method mix chosen to mirror the recall hosts. Eligibility required a manuscript and a code directory, excluded packages containing Python, and did not depend on RBD results.

The run produced 461 findings, each independently labeled by two human reviewers at the Social Catalyst Lab and ETH Zurich, with disagreements resolved by consensus. Of the 461 findings, 449 are valid defects, 6 fall outside scope, 4 are false alarms, and 2 are unclear. A valid defect is supported by the package evidence and falls within the verification task; a finding outside scope identifies an issue outside that task; a false alarm is not supported by the evidence; and an unclear finding cannot be resolved from the available package. The observed share classified as valid defects is 97.4% (95% Wilson interval: 95.5–98.5%), and the share not classified as valid is 2.6%.

Reviewers agreed on 91.3% of findings before consensus. Consensus exceeded both individual reviewers because jointly inspected ‘unclear’ cases resolved to valid. Validity is high in most chapters: all 141 execution findings are valid, the two consistency chapters sit at 96–98%, and data integrity is lower: 11 of 14 findings are valid, but with so few findings the estimate is imprecise. High- and low-confidence findings are valid at nearly the same rate (97% and 98%), so a finding's stated confidence says essentially nothing about whether it is valid. This mirrors the calibration results above.

To shed light on the basic economics of verification, we log human reviewers' time use, apply simple labor-cost accounting, and compare it with automated verification through back-of-the-envelope calculations. Dual human review runs at 5.2 minutes per finding, roughly 24 minutes per adjudicated paper. For an hourly human labor cost of $100 per hour, this corresponds to about $40 of expert time per paper. This can be benchmarked against $0.82 for the machine scan it audits ($0.007 at the cheap end of the frontier). These numbers price one specific task: an expert checking the machine's findings. They say nothing about the cost of verifying a paper from scratch, which takes hours. Adjudication time also tracks the difficulty ordering of the recall benchmark: execution findings take 6 to 12 seconds to confirm, while consistency findings take two to three minutes.

The precision estimate also helps interpret the findings on mutated papers that do not match the injected error. We did not adjudicate these, but the same configuration's findings on unmutated papers are 97% valid, so most of them are likely real defects of the host papers.

We also apply an automated verifier to the full 1,000-paper APE corpus. We will release that evaluation soon. It will report the scan design, findings, and interpretation; we do not use the controlled benchmark or precision exercise to impute unobserved corpus-wide error rates here.

In this paper, we developed CRED, a versioned classification of research errors and defects, designed to verify the verifiers: to measure how reliable automated, LLM-powered verification systems actually are. Using injected errors for which we know the ground truth, we find that publicly available off-the-shelf verifiers reach 99% detection recall. Closed-weight models lead, strictly speaking, but open-weight models have essentially caught up. We also show that ensembles of multiple inexpensive open-weight models are highly cost-effective. While we focus on recall, the most important dimension of reliability, it is not the whole story: reliability spans several properties that move independently — attribution, consistency across runs, calibration of confidence — and calibration in particular remains unsolved. We also show that when a paper contains more than one error, detection recall degrades. If autonomously generated papers produce many errors, we would therefore expect one-shot verifiers to struggle. In that case it may be necessary to iterate between verification and generation, reducing errors round by round until they have been eliminated. We aim to explore that in future research.²⁶26. It is worth noting that even if end-to-end autonomous policy evaluation is not the goal, or is viewed as currently unreliable, quantifying the reliability of the verifier helps with human-in-the-loop triage. On algorithmic triage and learning to defer to an expert: Raghu et al. (2019); Mozannar & Sontag (2020), ICML. Agarwal, Moehring & Wolitzky (2025) derive a sufficient statistic for deciding which decisions to automate and which to hand to a human.

We argue for the ideal that anything that can be verified should be verified. In practice, it is not obvious what that entails, and it raises a deeper question: where is the boundary between what is verifiable and what is not? At its core, a policy evaluation follows a particular formula. There is a policy of interest (a minimum wage increase in Pennsylvania) and one or more outcomes (employment). There is an estimand — the object of interest, such as the average effect of the increase on employment among fast-food workers. And there is an estimator applied to data, producing an estimate. What here is verifiable? The estimand is not, because it is a definition, not a claim — it is stated by the researcher, the policymaker, or an autonomous agent. There is nothing to check. The estimator rests on assumptions about the underlying data-generating process. For example, a difference-in-differences regression assumes parallel trends. This is a counterfactual assumption that cannot be verified, again by definition. Just as omitted variables are unobservable, so is the true error structure. For example, whether the DGP has homoskedastic errors is not a verifiable fact, even if, under additional assumptions, statistical tests can probe it. In each case, the boundary between what is fundamentally observable and unobservable determines what can be verified. But below that boundary lies a large and crucial territory of fact: the end-to-end execution of the research, from data fetching to estimation to the generation of tables to the prose that describes the process, its choices, and its output. In that territory, errors have a ground truth, and we argue that a reliable verifier should always detect them.

CRED is a first attempt at putting structure on that territory. Its four chapters sit entirely on the verifiable side of the boundary: whether the data match their claimed source, whether the code runs and reproduces its own output, whether the prose matches the code, and whether the prose matches the tables. Together with human adjudication, it allows us to quantify how reliable verifiers are. We claim neither that this classification is exhaustive nor that the errors we inject are representative of the errors AI-generated research will actually produce, now or in the future. Indeed, the reason we version the classification is that we expect it to evolve. It can expand vertically, into more refined sub-chapters, and horizontally, into new chapters. Moreover, we aim to explore in future research harder, more complex errors within existing chapters.

The value of a good verification system goes well beyond checking a single paper. One can improve the very harness that generated the research in the first place. How to build better generation harnesses for policy evaluation is an open research question. How to close the loop — creating the environment for a self-improving harness that learns to generate few or no errors — is, we believe, the most promising avenue toward autonomous policy evaluation at scale.

v0.92 (current). 100 host papers that passed RBD, one or two injected errors per instance, deterministic matcher, numeric confidence. All results on this page are scored under v0.92. We anticipate the codebook to evolve over time to reflect a wider breadth of error types found in automated policy evaluation; when it does, results under the new version will appear here as a dated entry, older versions will remain addressable, and numbers will not be compared across versions. The admission rule is fixed: a class enters the codebook only if there is a deterministic rule for how the error shows up in the paper and its code. A larger codebook may lower measured recall. Comparisons should therefore be made within a fixed benchmark version.

Rolling updates. CRED is a living benchmark. We update this page on a rolling basis as new models, effort settings, and completed run designs become available. In the coming days and weeks, we will add newly released models and complete additional AB and replicate runs. New configurations generally enter the capability sample first; interference and repeatability results follow when the additional runs are complete. We aim to run arms A and B for every eligible new model. We prioritize AB and replicate runs for models that enter the cost-recall frontier, major new flagship models, and configurations needed to complete effort ladders. The export date at the top of the page identifies the current results status. Additions of models or runs retain the existing benchmark version; changes to the benchmark, scoring rules, or error corpus require a new version. We will update the human-adjudication exercise as detector selection changes. We will release our evaluation of the 1,000-paper corpus soon.