AI may soon be capable of producing rigorous economic research. If that happens, policy evaluation could scale dramatically around the world: identifying what works, what fails, and what causes harm. To that end, we set up Project APE at the Social Catalyst Lab, led by Prof. David Yanagizawa-Drott, as an experiment testing whether an autonomous system can reliably generate, replicate, and verify empirical policy research. Everything remains public: papers, code, data, and failures. For a 3D visualisation of the project's global scope, click here. Project APE has also been by invitation at research and policy institutions internationally.
⚠️ Warning: We are learning how to build a reliable, autonomous research system. Expect bugs, errors, hallucinations, and trashable papers. None of the generated papers have been peer-reviewed and should not be used for evidence-based policy making. What does "autonomous" even mean?
How reliable is automated verification—and at what cost?
Automated verification means an AI reads a finished research paper—its data, code, and claims—and tries to catch the errors in it, the way a careful referee would. The animation follows that capability's cost–recall frontier from 2024 to the latest benchmark release: each point is a tested verifier configuration on CRED's controlled error-injection benchmark, and the frontier moves whenever a model or harness delivers more recall at lower cost. For examples of how errors are missed by specific verifiers, see these reports on Gemini 3.8 Flash, Sonnet 5, and GPT-4o mini.
From January to April 2026, Project APE generated its first 1,000 papers and then paused production to evaluate the system. We then developed the CRED taxonomy to classify errors and defects in empirical research. In Verifying the Verifiers, the working paper released in July 2026, we used it to create a controlled test by inserting known errors into 100 papers, then measured how reliably—and at what cost—different AI verifiers found them. The rankings below preserve the automated tournament as it stood when production paused.
02
CRED evaluation in progress
In a separate sample of 100 unaltered papers, human reviewers agreed that 97% of the selected verifier's reported findings were genuine errors or defects. Based on that verifier's reports, we estimate that every paper contains errors or defects under Project APE's CRED taxonomy—often several, including some classified as severe. These are automated findings; humans did not check all 1,000 papers individually. We are still analyzing the results and writing the report.
We are now prototyping an experiment with a second set of 1,000 paper ideas, testing whether and how the system can learn to improve itself.
The loop and its output. The loop itself ran without human input; the harness around it was re-engineered throughout — paper formats were deliberately varied, from longer and more ambitious to shorter no-revision runs, and a two-model co-authored variant with both Claude Code and Codex CLI joined the solo pipeline. Review in this era meant automated LLM referees — like journal referees, they read the paper but could not run the code — and they were themselves unverified. Independent verification (CRED) came after the freeze.
The first 1,000 papers
LLM-as-a-Judge Tournament
The leaderboard below records the first 1,000 papers as they stood when production was frozen for evaluation. It reflects a noisy triage signal from an LLM judge that read only each paper's PDF and assessed it as an economics journal editor might. It is therefore not peer review, a CRED-based score, or evidence that a paper is correct. Because the judge could not inspect code or data, it could not systematically assess the papers' underlying validity. We are now writing up findings from a separate CRED-based analysis of errors and defects in these papers. Future experiments will update both the leaderboard and how it is constructed, including by incorporating CRED-based verification.
How the Tournament Worked
▼
The Tournament
Papers competed in head-to-head matches. Each day at 9:00 AM UTC, we ran a round of matches. An LLM judge read both papers and picked one as preferred (or declared a tie). The preferred paper gained rating points; the other lost points. Over time, papers consistently favored by the LLM judge rose in the rankings—though this was a noisy and potentially biased signal, not ground truth.
How Matches Work
For each match, we sent both paper PDFs directly to Gemini 3.1 Flash Lite (Google's LLM). The model saw the full papers—text, figures, tables, and formatting.
The judge was prompted to act as a "senior editor at a top economics journal" and evaluate papers on: identification strategy (is the causal inference credible?), novelty, policy relevance, execution quality, and appropriate scope. .
To control for position bias (LLMs sometimes prefer whichever paper they see first), we ran each comparison twice with the papers swapped. A paper must win both rounds to win the match; otherwise it's a tie.
Rating System
Rankings use TrueSkill, a Bayesian rating system developed by Microsoft. Each paper has two numbers:
μ (mu) — estimated skill level. Higher = paper wins more often.
σ (sigma) — uncertainty. Decreases as the paper plays more matches.
Papers are ranked by their conservative rating (μ − 3σ), which represents the lower bound of estimated skill. This means papers need consistent wins across multiple matches — a single lucky win won't send a paper to the top.
Head-to-Head Statistics
Win counts exclude 967 ties where the judge couldn't determine a clear winner.
Prob(Human Win) answers: if we randomly pick one human paper and one APE paper, what's the probability the human paper wins according to the LLM judge? Computed from TrueSkill ratings, accounting for uncertainty — papers with fewer matches contribute less certainty. The main metric compares recent cohorts (last 25 of each); all-time (88.1%) includes all 990 AI and 43 human papers with 5+ matches.
Matchup Selection
Each day we ran 50 matches (100 LLM calls with position swapping) in 10 batches of 5. Within each batch, no paper played twice. We combined random matching with structured matching.
Important Caveats ⚠️
The ⚠️ warning icon in the leaderboard indicates AI-generated papers that have not been peer reviewed. The LLM judge is not a substitute for human peer review. AI-generated papers may contain errors, hallucinations, or fabricated results. The subsequent automated CRED evaluation reported errors or defects across the full 1,000-paper corpus. Read the evaluation context.
The judge evaluates the PDF only, not the underlying code or data. Rankings should not be taken at face value. That's why everything is open source — code, data, and papers are all public so anyone can spot errors, report issues, and contribute improvements.
Ranking Metrics
▼
Rank
Position based on conservative rating (lower bound of estimated skill)
Total tokens used for tournament (excludes paper generation tokens): 1,476,570,316
Papers competed in head-to-head matchups judged by Gemini 3.1 Flash Lite, prompted as a senior editor at a top economics journal. Each comparison used position-swapping to control for bias. Rankings are a noisy and potentially biased signal of quality, not ground truth.
