ROI of GenAI in the SDLC: Related Research

Table of contents

Sema’s AI Code Monitor now helps Engineering leaders estimate the Return on Investment (ROI) from implementing GenAI coding tools.

To assist Engineering leaders in this effort, we have collected related research in the following areas:

Estimates of Time that Developers are Coding vs. Non Coding
Potential Productivity Improvements: New Codebase
Potential Productivity Improvements: Existing Codebase- Primarily Tech Debt Reduction
Potential Productivity Improvements: Existing Codebase- Primarily new Features and Functionality
Potential Productivity Improvements: Existing Codebase- Primarily Bug Fixes
Potential Productivity Improvements: Existing Codebase- Primarily Prototyping
Other Relevant Research

1. Estimates of Time that Developers are Coding vs. Non Coding

12.5%. Software Study: "Global Code Time Report" [Based on data from 250K+ developers in our global community, developers code 52 minutes per day — about 4 hours and 21 minutes during a normal workweek from Monday to Friday.] January, 2022. https://www.software.com/reports/code-time-report

43% Globe News Wire Survey: "Developer Shortage Remains the Top Challenge Two Years in a Row, 2023 Reveal Annual Software Developer Survey finds" [While nearly all developers (88.8%) work with a designer or design team, nearly half of a developer’s time (43.4%) is spent on coding an app, the survey found.] February 23, 2023. https://www.globenewswire.com/news-release/2023/02/23/2614440/0/en/Developer-Shortage-Remains-the-Top-Challenge-Two-Years-in-a-Row-2023-Reveal-Annual-Software-Developer-Survey-finds.html

12.5%. Software.com: “2023 Future of Work Study” [Developers are spending marginally less time coding per weekday — 59.9 minutes in 2023 compared to 64.2 minutes in 2020 (-7%)] 2023. https://www.software.com/reports/future-of-work

2. Potential Productivity Improvements: New Codebase

50% improvement. McKinsey Study: “Unleashing developer productivity with generative AI” [Documenting code functionality for maintainability (which considers how easily code can be improved) can be completed in half the time, writing new code in nearly half the time,] June 27, 2023. https://www.mckinsey.com/capabilities/mckinsey-digital/our-insights/unleashing-developer-productivity-with-generative-ai

80% improvement. Sema’s internal estimates based on our Engineers. Note this estimate is heavily weighted towards building a commercial grade working prototype. Less-than-commercial grade working prototype can be 95% faster. Commercial grade General Availability product is likely 60% faster. Contact ai@semasoftware.com if you would like to discuss.

3. Potential Productivity Improvements: Existing Codebase- Primarily Tech Debt Reduction

66% improvement. McKinsey Study: "Unleashing developer productivity with generative AI" Documenting code functionality for maintainability (which considers how easily code can be improved) can be completed in half the time, writing new code in nearly half the time, and optimizing existing code (called code refactoring) in nearly two-thirds the time (Exhibit 1). ] June 27, 2023. https://www.mckinsey.com/capabilities/mckinsey-digital/our-insights/unleashing-developer-productivity-with-generative-ai.

73% improvement. Slalom consultancy experiment [73% faster migration to modern programming languages and 3x faster migration to AWS Cloud Development Kit(CDK)—note only the 73% was included] April 1, 2024. https://medium.com/slalom-technology/bridging-the-gap-maximizing-developer-productivity-with-generative-ai-02967e8f115a

70% reduction. Gartner Study: “Gartner's Top Strategic Predictions for 2024 and Beyond” [By 2027, GenAI tools will be used to explain legacy business applications and create appropriate replacements, reducing modernization costs by 70%.] December 04, 2023. https://www.gartner.com/en/articles/gartner-s-top-strategic-predictions-for-2024-and-beyond

4. Potential Productivity Improvements: Existing Codebase- Primarily new Features and Functionality

30% acceptance rate. Github Copilot Study: “The economic impact of the AI-powered developer lifecycle and lessons from GitHub Copilot”[On average, within the first year in the market, users accept nearly 30%of code suggestions from GitHub Copilot and report increased productivity from these acceptances.] June 27, 2023. https://github.blog/news-insights/research/the-economic-impact-of-the-ai-powered-developer-lifecycle-and-lessons-from-github-copilot/

5. Potential Productivity Improvements: Existing Codebase- Primarily Bug Fixes

6. Potential Productivity Improvements: Existing Codebase- Primarily Prototyping

7. Other Relevant Research

Princeton, MIT, Microsoft Research: “The Effects of Generative AI on High Skilled Work: Evidence from Three Field Experiments with Software Developers” [26% decrease in task completion time], September 5, 2024. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4945566