New study raises questions about true impact of AI coding assistants

AI coding assistants: Promise vs. reality: Recent studies and developer experiences reveal a mixed picture of the impact of AI-powered coding tools on productivity and code quality.

• A study by Uplevel, comparing the output of 800 developers using GitHub Copilot over a three-month period to their previous performance, found no significant improvements in productivity metrics.
• The study measured pull request (PR) cycle time and PR throughput, finding no substantial gains for developers using Copilot.
• Surprisingly, the use of GitHub Copilot was associated with a 41% increase in bug introduction.
• That said, there is significant evidence that other developers and organizations have seen marked improvements to productivity and output.

Developer burnout and workload: Contrary to expectations, coding assistants and AI developer tools have not significantly alleviated developer burnout or improved work-life balance according to the study.

• The Uplevel study found that the amount of working time spent outside standard hours decreased for both the control group and the test group using Copilot.
• However, the decrease in non-standard working hours was more pronounced for developers not using Copilot.

Industry perspectives and experiences: Developers and companies report varying outcomes when using AI coding assistants, highlighting the complexity of integrating these tools into existing workflows.

• Ivan Gekht, CEO of Gehtsoft USA, notes that his company hasn’t seen major productivity gains with coding assistants based on large language models (LLMs).
• Gekht emphasizes that software development is primarily a cognitive process, with actual coding being a smaller part of the job.
• In contrast, Travis Rehl, CTO of Innovative Solutions, reports significant productivity increases of two to three times when using tools like Claude Dev and GitHub Copilot.

Challenges and limitations: The implementation of AI coding assistants comes with several hurdles that developers and organizations need to navigate.

• Gehtsoft USA observed that AI-generated code can become less consistent when different parts are developed using various prompts.
• Debugging and troubleshooting AI-generated code can become increasingly challenging, sometimes making it more efficient to rewrite the code from scratch.
• The lack of critical thinking and self-awareness in LLMs limits their ability to fully replace human developers in complex software development tasks.

Potential benefits and use cases: Despite the challenges, AI coding assistants show promise in certain areas of software development.

• These tools can be effectively used for quickly substituting code or optimizing code paths by reworking segments.
• Rehl’s team at Innovative Solutions completed a customer project in 24 hours using coding assistants, a task that would have previously taken about 30 days.

Managing expectations: Industry experts emphasize the importance of setting realistic expectations for AI coding assistants and their role in software development.

• Claims of coding assistants replacing entire development teams are considered unrealistic by many in the industry.
• Rehl suggests that expectations should be tempered, as coding assistants won’t write all the code or produce entirely correct code on the first attempt.
• The effective use of these tools involves an iterative process that can potentially increase coding speed by two to three times when used correctly.

Looking ahead: Evolution of AI coding assistants: As the technology behind AI coding assistants continues to advance rapidly, their impact on software development may change.

• Uplevel’s study suggests that organizations should continue using coding assistants, acknowledging the rapid pace of technological advancement in this field.
• Future studies and real-world applications will likely provide more insights into the long-term effects of AI coding assistants on developer productivity and code quality.
• The software development industry may need to adapt its practices and metrics to fully leverage the potential of AI coding assistants while mitigating their current limitations.