Studies of real developers
Does it make developers faster?
Trials that gave some developers AI and not others. The answers differ by who was tested, on what, and with which tools, and the studies run by companies that make the tools report the biggest gains.
+20%how much faster experienced developers believed AI made them
−19%what was measured: they took 19% longer with AI
METR now calls these 2025 results out of date. Its 2026 follow-up points to a speedup but, in its words, gives “an unreliable signal”. METR, 2025, Jul 10, 2025 · METR, Feb 2026, Feb 24, 2026
randomized trial follow-up
18%faster in METR's newer estimate, which it calls an unreliable signal
METR
“an unreliable signal of the current productivity effect of AI tools”
- “developers choosing not to participate in the study because they do not wish to work without AI”
- “developers are more sped up from AI tools now”
- “only very weak evidence for the size of this increase”
Original developers, later study: “we now estimate a speedup of -18%”
METR writes the result with the same sign as in 2025, where +19% meant slower, so −18% means faster. Its range runs from 38% faster to 9% slower.
METR uplift study update · METR, Feb 24, 2026
randomized controlled trial
19%longer to finish real tasks with AI, for experienced developers in early 2025
METR
“When developers are allowed to use AI tools, they take 19% longer to complete issues”
- “expected AI to speed them up by 24%”
- “they still believed AI had sped them up by 20%”
Early-2025 AI tools, in the developers' own code.
Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity · METR, Jul 10, 2025
three randomized field experimentsCompany-reported
+26%more tasks completed by developers given GitHub Copilot
Zheyuan (Kevin) Cui, Mert Demirer, Sonia Jaffe, Leon Musolff, Sida Peng, Tobias Salz
“a 26.08% increase (SE: 10.3%) in completed tasks among developers using the AI tool.”
Microsoft researchers, with Copilot at Microsoft, Accenture and a large US company, using 2022 and 2023 tools. SE is the margin of error.
The Effects of Generative AI on High-Skilled Work: Evidence from Three Field Experiments with Software Developers · SSRN 4945566, Aug 21, 2025
randomized controlled trialCompany-reported
~21%less time on a set task, for Google's own engineers
Elise Paradis, Kate Grey, Quinn Madison, Daye Nam, Andrew Macvean, Vahid Meimand, Nan Zhang, Ben Ferrari-Church, Satish Chandra (Google)
“stands at about 21%, although our confidence interval is large.”
Google's own engineers and tools, 2024. The authors say the margin of error is large.
How much does AI impact development speed? An enterprise-based randomized controlled trial · arXiv:2410.12944 (v3), Nov 11, 2024
randomized controlled experimentCompany-reported
55.8%faster at one set task with GitHub Copilot
Sida Peng, Eirini Kalliamvakou, Peter Cihon, Mert Demirer (GitHub / Microsoft Research co-authors)
“The treatment group, with access to the AI pair programmer, completed the task 55.8% faster than the control group.”
2023, by GitHub and Microsoft researchers.
The Impact of AI on Developer Productivity: Evidence from GitHub Copilot · arXiv:2302.06590, Feb 13, 2023
