Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity

★

☆

SHARE

Overview

Paper Summary

Conflicts of Interest

Identified Weaknesses

Rating Explanation

Good to know

Topic Hierarchy

File Information

Paper Summary

Paperzilla title

Experienced Open-Source Developers Slowed Down by AI...But Think It Speeds Them Up!

In a study of experienced open-source developers, access to AI tools like Cursor Pro and Claude actually *increased* the time it took to complete tasks by 19%. Surprisingly, both expert forecasters and the developers themselves predicted significant *speedups* from using AI, even after the developers had used the tools. This highlights a disconnect between perceived and actual AI impact on productivity.

Possible Conflicts of Interest

None identified.

Identified Weaknesses

Generalizability

The study focuses on experienced developers working on mature, complex projects, which may not be representative of all software development settings. Results may differ for newer projects, less experienced programmers, or different types of tasks.

AI Tool Limitations

The study's timeframe (early 2025) means the AI tools used may already be outdated given rapid AI advancements. The observed slowdown might not apply to more advanced future systems.

Experimental Artifacts

The non-blinded design and self-reported time tracking could have introduced biases, like over-reliance on AI due to the experiment's focus or unintentional changes in work habits due to being observed.

Limited Statistical Power

While the study analyzes multiple potential contributing factors, the sample size may be insufficient for definitive conclusions about the individual impact of each factor.

Rating Explanation

While the setting-specific nature limits generalizability, the study uses a strong experimental design (RCT) with real-world tasks and a relevant developer population, offering valuable insights into the current limitations of AI in boosting productivity for this specific group. The rich qualitative and quantitative data further strengthens the analysis, offering compelling explanations for the unexpected slowdown.

Good to know

This is our free standard analysis. Paperzilla Pro fact-checks every citation, researches author backgrounds and funding sources, and uses advanced AI reasoning for more thorough insights.

Explore Pro →