AI agents can now write substantial amounts of research code. In a recent self-selected survey, 39 percent of economists reported using coding agents more than once a week. As tools such as Claude Code and GitHub Copilot become more widely available, producing code is getting easier. But who is reviewing what the agents produce?
In our earlier tests, agents demonstrated their ability to solve coding tasks, scoring between 77 and 81 percent on the assessment we use to hire research assistants. Yet their code was complex, and the rationale for their analytical decisions was unclear. Another example is a recent benchmark of econometric tasks, letting AI agents run and revise their own code raised the share of correct results from 79 to 97 percent, but every remaining failure was silent: the code ran and returned wrong numbers. But code that runs is not necessarily code a team can work with. Research projects often last several years, and the code needs to be understandable to whoever picks it up next. It also needs to meet team standards, such as documenting analytical decisions and validating merges.
As more journals require researchers to share data and code, research code is receiving more scrutiny. And that scrutiny has exposed two different problems: code may fail to reproduce the published results, and even when it does reproduce them, it may still be wrong. Only one in five packages in our checks reproduce as submitted. At the American Economic Association, just 31 of 404 manuscripts at conditional acceptance were recommended for acceptance without changes to their replication packages (Vilhuber 2025).
Reproducibility, however, does not imply the code is correct. An audit of 110 articles in leading economics and political-science journals found substantive coding errors in 26, even though 85 percent were computationally reproducible. Errors included duplicate observations, incorrectly coded treatment variables, and models that differed from those described in the papers.
AI thus poses a challenge, and offers an opportunity. By greatly expanding research code production, it makes systematic review more important. Yet these tools can also put careful code review within reach of researchers who lack expert coding skills or a research assistant or colleague to check their work.
From manual peer review to review on demand
Before AI coding tools became available, our DECDI team organized structured peer code reviews. Colleagues reviewed each other’s code using standard checklists. Participants reported learning something new even in repeat rounds; one described the checklists as “very helpful starting points for the review process.” About 15 people took part each quarter, but finding time and coordinating reviews limited engagement. Since then, we have combined these checklists with lessons from reproducibility reviews into a reusable AI skill. It applies the same checks a peer reviewer would, but researchers can run it whenever their code is ready, without waiting for a review round.
Why use a skill rather than a prompt?
When you ask an agent to “review my code”, it can produce different results depending on the model, the type of errors it checks for, and the depth of its review. A skill, on the other hand, makes the review more consistent because the agent follows the same steps each time. It runs the same checks, uses the same reference material, and produces the report in the same format every time.
