What is this?

We built an AI system (APE) that writes economics research papers from scratch — finding data, running statistical analysis, and producing full academic papers. At its core, this is currently powered by Claude Code, with input from GPT 5.2 via API calls for additional diversity and entropy. This leaderboard tracks whether those AI-written papers can "compete" with human-written papers from top economics journals, through an automated process.

The Tournament

Papers compete in head-to-head matches. Each day at 9:00 AM UTC, we run a round of matches. An LLM judge reads both papers and picks one as preferred (or declares a tie). The preferred paper gains rating points; the other loses points. Over time, papers consistently favored by the LLM judge rise in the rankings — though this reflects LLM preferences, not necessarily true quality.

How Matches Work

For each match, we send both paper PDFs directly to Gemini 3 Flash (Google's LLM). The model sees the full papers — text, figures, tables, formatting — exactly as a human reviewer would.

The judge is prompted to act as a "senior editor at a top economics journal" and evaluate papers on: identification strategy (is the causal inference credible?), novelty, policy relevance, execution quality, and appropriate scope.

To control for position bias (LLMs sometimes prefer whichever paper they see first), we run each comparison twice with the papers swapped. A paper must win both rounds to win the match; otherwise it's a tie. All matches can be found here.

Rating System

Rankings use TrueSkill, a Bayesian rating system developed by Microsoft for Xbox matchmaking. Each paper has two numbers:

• μ (mu) — estimated skill level. Higher = paper wins more often.
• σ (sigma) — uncertainty. Decreases as the paper plays more matches.

Papers are ranked by their conservative rating (μ − 3σ), which represents the lower bound of estimated skill. This means papers need consistent wins across multiple matches — a single lucky win won't send a paper to the top.

Head-to-Head Statistics

Win counts exclude 12 ties where the judge couldn't determine a clear winner.

Prob(Human Win) answers: if we randomly pick one human paper and one AI paper, what's the probability the human paper wins according to the LLM judge? Computed from TrueSkill ratings, accounting for uncertainty — papers with fewer matches contribute less certainty. The main metric compares recent cohorts (last 25 of each); all-time (94.3%) includes all 54 AI and 43 human papers with 5+ matches.

Matchup Selection

Each day we run 50 matches (100 LLM calls with position swapping) in 10 batches of 5. Within each batch, no paper plays twice.

80% of matches use information-gain sampling — we prioritize pairs where both papers have high rating uncertainty and similar skill levels, maximizing what we learn from each expensive LLM call. Of these, half are constrained to cross-type (one AI paper vs one human paper), ensuring we continuously compare AI and human papers rather than just ranking within each group. The remaining 20% are random for exploration and robustness.

This design is cost-efficient: LLM judge calls are expensive, so we extract maximum signal per match. Papers with low uncertainty or extreme ratings may not match every day — that's by design.

Important Caveats ⚠️

The ⚠️ warning icon in the leaderboard indicates AI-generated papers that have not been peer reviewed. The LLM judge is not a substitute for human peer review. AI-generated papers may contain errors, hallucinations, or fabricated results. In fact, we have found that these are very common and sometimes take a lot of effort to spot. If a paper looks too good to be true, it probably is.

The judge evaluates the PDF only, not the underlying code or data. Rankings should not be taken at face value. That's why everything is open source — code, data, and papers are all public so anyone can spot errors, report issues, and contribute improvements.