Judge Calibration
Testing the LLM referee on papers accepted by top economics journals
How Calibration Works
▼Why Calibrate?
When an AI-generated paper gets rejected or "accepted" by our system, does that mean anything? Or is it all noise? Calibration answers this question. We run working paper versions of research that humans validated — accepted at AER and AEJ:Policy — through the exact same LLM review process. If the judge correctly recognizes these as publication-quality, its evaluations of AI papers carry weight. If it can't distinguish validated human work from noise, neither can we trust its AI paper judgments.
Method
- Same prompt: Identical referee prompt used for AI papers — no special treatment
- 10 reviews per paper: Measures both typical judgment and variance
- 5 decisions: Accept, Cond. Accept, Minor Rev (accepting) vs Major Rev, Reject
Good Calibration
For papers that top journals accepted: high acceptance rate (Accept/Cond. Accept/Minor Rev), few rejections. Variance across reviews (temperature=0.5) provides signal on decision confidence.
Caveats
- Working papers, not final versions (may have minor issues fixed before publication)
- Single model (Gemini 3 Flash) — other models may have different biases
- Results depend on the specific prompt version
Latest Results
92%
Acceptance Rate
Accept + Cond. Accept + Minor Rev
90.7%
Majority Agreement
At least 2/3 reviews agree
Overall Decision Distribution
Per-Paper Results
Click any paper to see its full decision distribution. Each paper was reviewed 10 times to measure consistency.
AEJ:PolicyA Matter of Time? Measuring Effects of Public Schooling Expansions on Families
Cond. Accept
AEJ:PolicyAdjustable Product Attributes, Indirect Network Effects, And Subsidy Design: The Case of Electric Vehicles
Cond. Accept
AEJ:PolicyAre Complementary Policies Substitutes? Evidence from R&D Subsidies in the UK
Cond. Accept
AEJ:PolicyBlack Lives: The High Cost of Segregation
Minor Rev
AEJ:PolicyCan You Erase the Mark of a Criminal Record? Labor Market Impacts of Criminal Record Remediation
Cond. Accept
AEJ:PolicyChildhood Health Shocks and the Intergenerational Transmission of Inequality
Minor Rev
AEJ:PolicyChina's Nationwide CO2 Emissions Trading System: A General Equilibrium Assessment
Minor Rev
AEJ:PolicyCollective Bargaining Rights, Policing, and Civilian Deaths
Minor Rev
AEJ:PolicyCooking, Health, and Daily Exposure to Pollution Spikes
Cond. Accept
AEJ:PolicyCross-State Strategic Voting
Cond. Accept
AEJ:PolicyEstimating the Economic Value of Zoning Reform
Cond. Accept
AEJ:PolicyFemale Leaders and Intrahousehold Dynamics: Evidence from State Elections in India
Minor Rev
AEJ:PolicyFiscal Procyclicality in Commodity Exporting Countries: How Much Does It Pour and Why?
Major Rev
AEJ:PolicyFlood Risk Mapping and the Distributional Impacts of Climate Information
Cond. Accept
AEJ:PolicyHacked to Pieces? The Effects of Ransomware Attacks on Hospitals and Patients
Major Rev
AEJ:PolicyHarnessing Deductions to Increase Tax Compliance and Formalization
Minor Rev
AEJ:PolicyHurricanes, Climate Change Policies and Electoral Accountability
Major Rev
AEJ:PolicyInvisible Wounds: How Mental Disability Benefits Shape Veteran Well-Being
Accept
AEJ:PolicyPerils of the Paperwork: The Impact of Information and Application Assistance on Social Benefit Take-Up in India
Major Rev
AEJ:PolicyPlata y Plomo: How Higher Wages Expose Politicians to Criminal Violence
Cond. Accept
AEJ:PolicyPolling Place Location and the Costs of Voting
Minor Rev
AEJ:PolicyPolluted IPOs
Minor Rev
AEJ:PolicyPunishing Financial Crimes: The Impact of Prison Sentences on Defendants and Their Colleagues
Cond. Accept
AEJ:PolicyShort- and Long-Term Effects of Universal Preschool: Evidence from the Arab Population in Israel
Minor Rev
AEJ:PolicyState Recreational Cannabis Laws and Racial Disparities in the Criminal Legal System
Minor Rev
AEJ:PolicyTax and Occupancy of Business Properties: Evidence from UK Business Rate Reliefs
Minor Rev
AEJ:PolicyThe Effect of Deactivating Facebook and Instagram on Users' Emotional State
Accept
AEJ:PolicyThe Option Value of Municipal Liquidity
Minor Rev
AEJ:PolicyThe Pass-Through of Retail Crime
Minor Rev
AEJ:PolicyThe Unintended Consequences of Academic Leniency
Cond. Accept
AEJ:PolicyThe Value of School Social Climate Information: Evidence from Chicago Housing Transactions
Minor Rev
AEJ:PolicyThe Welfare Effects of Eligibility Expansions: Theory and Evidence from SNAP
Cond. Accept
AEJ:PolicyTrade Protection Along Supply Chains
Cond. Accept
AEJ:PolicyVertical Integration and Cream Skimming of Profitable Referrals: The Case of Hospital-Owned Skilled Nursing Facilities
Cond. Accept
AERAbundance from Abroad: Migrant Income and Long-Run Economic Development
Accept
AERDynamics of the Long Term Housing Yield: Evidence from Natural Experiments
Accept
AERImmigration, Innovation, and Growth
Accept
AERMarket Power and Capital Constraints
Minor Rev
AERTemporary Layoffs, Loss-of-Recall and Cyclical Unemployment Dynamics
Minor Rev
AERThe Value of Clean Water: Experimental Evidence from Rural India
Cond. Accept
AERWhy is Workplace Sexual Harassment Underreported? The Value of Outside Options Amid the Threat of Retaliation
Cond. Accept
AERWork From Home and the Office Real Estate Apocalypse
Minor Rev
AERZero-Sum Thinking and the Roots of US Political Differences
Minor Rev