Judge Calibration
Testing the LLM referee on papers accepted by top economics journals
How Calibration Works
▼Why Calibrate?
When an AI-generated paper gets rejected or "accepted" by our system, does that mean anything? Or is it all noise? Calibration answers this question. We run working paper versions of research that humans validated — accepted at AER and AEJ:Policy — through the exact same LLM review process. If the judge correctly recognizes these as publication-quality, its evaluations of APE papers carry weight. If it can't distinguish validated human work from noise, neither can we trust its APE paper judgments.
Method
- Same prompt: Identical referee prompt used for APE papers —
- 10 reviews per paper: Measures both typical judgment and variance
- 5 decisions: Accept, Cond. Accept, Minor Rev (accepting) vs Major Rev, Reject
Good Calibration
For papers that top journals accepted: high acceptance rate (Accept/Cond. Accept/Minor Rev), few rejections. Variance across reviews (temperature=0.5) provides signal on decision confidence.
Caveats
- Working papers, not final versions (may have minor issues fixed before publication)
- Single model (Gemini 3 Flash) — other models may have different biases
- Results depend on the specific prompt version
Latest Results
94.5%
Acceptance Rate
Accept + Cond. Accept + Minor Rev
65.1%
Majority Agreement
At least 2/3 reviews agree
Overall Decision Distribution
Per-Paper Results
Click any paper to see its full decision distribution. Each paper was reviewed 10 times to measure consistency.
AEJ:PolicyA Matter of Time? Measuring Effects of Public Schooling Expansions on Families
Minor Rev
AEJ:PolicyAdjustable Product Attributes, Indirect Network Effects, And Subsidy Design: The Case of Electric Vehicles
Cond. Accept
AEJ:PolicyAre Complementary Policies Substitutes? Evidence from R&D Subsidies in the UK
Accept
AEJ:PolicyBlack Lives: The High Cost of Segregation
Accept
AEJ:PolicyCan You Erase the Mark of a Criminal Record? Labor Market Impacts of Criminal Record Remediation
Accept
AEJ:PolicyChildhood Health Shocks and the Intergenerational Transmission of Inequality
Accept
AEJ:PolicyChina's Nationwide CO2 Emissions Trading System: A General Equilibrium Assessment
Accept
AEJ:PolicyCollective Bargaining Rights, Policing, and Civilian Deaths
Accept
AEJ:PolicyCooking, Health, and Daily Exposure to Pollution Spikes
Accept
AEJ:PolicyCross-State Strategic Voting
Accept
AEJ:PolicyEstimating the Economic Value of Zoning Reform
Accept
AEJ:PolicyFemale Leaders and Intrahousehold Dynamics: Evidence from State Elections in India
Minor Rev
AEJ:PolicyFiscal Procyclicality in Commodity Exporting Countries: How Much Does It Pour and Why?
Major Rev
AEJ:PolicyFlood Risk Mapping and the Distributional Impacts of Climate Information
Accept
AEJ:PolicyHacked to Pieces? The Effects of Ransomware Attacks on Hospitals and Patients
Major Rev
AEJ:PolicyHarnessing Deductions to Increase Tax Compliance and Formalization
Accept
AEJ:PolicyHurricanes, Climate Change Policies and Electoral Accountability
Cond. Accept
AEJ:PolicyInvisible Wounds: How Mental Disability Benefits Shape Veteran Well-Being
Accept
AEJ:PolicyPerils of the Paperwork: The Impact of Information and Application Assistance on Social Benefit Take-Up in India
Accept
AEJ:PolicyPlata y Plomo: How Higher Wages Expose Politicians to Criminal Violence
Accept
AEJ:PolicyPolling Place Location and the Costs of Voting
Accept
AEJ:PolicyPolluted IPOs
Accept
AEJ:PolicyPunishing Financial Crimes: The Impact of Prison Sentences on Defendants and Their Colleagues
Accept
AEJ:PolicyShort- and Long-Term Effects of Universal Preschool: Evidence from the Arab Population in Israel
Accept
AEJ:PolicyState Recreational Cannabis Laws and Racial Disparities in the Criminal Legal System
Minor Rev
AEJ:PolicyTax and Occupancy of Business Properties: Evidence from UK Business Rate Reliefs
Minor Rev
AEJ:PolicyThe Effect of Deactivating Facebook and Instagram on Users' Emotional State
Accept
AEJ:PolicyThe Option Value of Municipal Liquidity
Accept
AEJ:PolicyThe Pass-Through of Retail Crime
Accept
AEJ:PolicyThe Unintended Consequences of Academic Leniency
Accept
AEJ:PolicyThe Value of School Social Climate Information: Evidence from Chicago Housing Transactions
Minor Rev
AEJ:PolicyThe Welfare Effects of Eligibility Expansions: Theory and Evidence from SNAP
Accept
AEJ:PolicyTrade Protection Along Supply Chains
Accept
AEJ:PolicyVertical Integration and Cream Skimming of Profitable Referrals: The Case of Hospital-Owned Skilled Nursing Facilities
Accept
AERAbundance from Abroad: Migrant Income and Long-Run Economic Development
Accept
AERDynamics of the Long Term Housing Yield: Evidence from Natural Experiments
Accept
AERImmigration, Innovation, and Growth
Accept
AERMarket Power and Capital Constraints
Accept
AERTemporary Layoffs, Loss-of-Recall and Cyclical Unemployment Dynamics
Accept
AERThe Value of Clean Water: Experimental Evidence from Rural India
Accept
AERWhy is Workplace Sexual Harassment Underreported? The Value of Outside Options Amid the Threat of Retaliation
Accept
AERWork From Home and the Office Real Estate Apocalypse
Accept
AERZero-Sum Thinking and the Roots of US Political Differences
Cond. Accept