← Back to papers

Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity

★ ★ ★ ★ ☆

Overview

Paper Summary › Explain Like I'm Five › Conflicts of Interest › Identified Limitations › Rating Explanation › Good to know › Topic Hierarchy › File Information ›

Paper Summary

Paperzilla title
Experienced Open-Source Developers Slowed Down by AI...But Think It Speeds Them Up!

In a study of experienced open-source developers, access to AI tools like Cursor Pro and Claude actually *increased* the time it took to complete tasks by 19%. Surprisingly, both expert forecasters and the developers themselves predicted significant *speedups* from using AI, even after the developers had used the tools. This highlights a disconnect between perceived and actual AI impact on productivity.

Explain Like I'm Five

AI tools meant to help computer programmers were found to slow them down, even though everyone thought they'd be faster. Programmers spent more time fiddling with the AI than actually coding.

Possible Conflicts of Interest

None identified.

Identified Limitations

Generalizability
The study focuses on experienced developers working on mature, complex projects, which may not be representative of all software development settings. Results may differ for newer projects, less experienced programmers, or different types of tasks.
AI Tool Limitations
The study's timeframe (early 2025) means the AI tools used may already be outdated given rapid AI advancements. The observed slowdown might not apply to more advanced future systems.
Experimental Artifacts
The non-blinded design and self-reported time tracking could have introduced biases, like over-reliance on AI due to the experiment's focus or unintentional changes in work habits due to being observed.
Limited Statistical Power
While the study analyzes multiple potential contributing factors, the sample size may be insufficient for definitive conclusions about the individual impact of each factor.

Rating Explanation

While the setting-specific nature limits generalizability, the study uses a strong experimental design (RCT) with real-world tasks and a relevant developer population, offering valuable insights into the current limitations of AI in boosting productivity for this specific group. The rich qualitative and quantitative data further strengthens the analysis, offering compelling explanations for the unexpected slowdown.

Good to know

This is the Starter analysis. Paperzilla Pro fact-checks every citation, researches author backgrounds and funding sources, and uses advanced AI reasoning for more thorough insights.

Explore Pro →

File Information

Original Title: Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity
Uploaded: September 18, 2025 at 08:06 AM
Privacy: Public