Table of contents
Executive Summary
This whitepaper analyzes six metrics collected from 600 codebases, written by 162,000 Developers and containing 4 billion lines of code, that Sema has analyzed through its comprehensive codebase scan methodology.
The dataset comes from codebase scans of software companies and tech-enabled organizations. The businesses include software and tech-enabled businesses, such as but not limited to eCommerce, EdTech, Fintech, Security, Manufacturing software, and non-software companies such as Consultancies and Distributors.
This whitepaper analyzes one metric or set of metrics from each of six functional areas that make up Codebase Health. Three of the functional areas relate to Product delivery risk (code quality, development process, team). Three relate to Compliance risk (cyber security, code security, and intellectual property risk from Open Source code), One important metric from each of the six functional areas is analyzed in this whitepaper.
Key findings include:
- organizations face significant compliance risk, including hidden risks that are not readily identifiable to Engineers without tooling, and
- a substantial variation in metrics was observed, greater within companies at similar stages of their lifecycle than between segments.
Potential implications are included for ongoing operations of software organizations as well as during mergers and acquisitions.
The whitepaper concludes with five future research questions, including cross-industry and intra-industry comparisons.
Methodology
This whitepaper analyzes the results of Sema’s 600 most recent organizational codebases analyzed. In total, the dataset includes:
- 42,000 repositories
- The work of 162,000 Developers
- 4 billion lines of code
- 17.2 billion bytes of code
These 600 organizations are grouped into eight segments based on their size and stage, from solo programmers to 5,000 person dev teams. Our ingoing hypothesis, partially confirmed and partially refuted by our research, is that companies at a similar lifecycle stage would have common codebase traits.The dataset contains only commercial organizations and covers the gamut of industries, not only software companies across 20+ verticals but also tech-enabled organizations such as consultancies, pharmaceutical distributors, and investment banks.
.svg)
The organizations in the data set are built by software developers supervised by the organization, whether in-house or via third-party development shops, for external and or internal uses. Organizations that rely solely on Commercial-off-the-shelf (COTS) software are not included.
This whitepaper analyzes six functional areas that collectively are part of a Codebase Health Framework.
Codebase Health Framework
The six metrics come from two main categories:
- Product Risk, affecting the ability to deliver code with the right requirements predictably and with suitable velocity.
- Compliance Risk, the degree to which the codebase is susceptible to security breaches and legal risk.
There is one metric analyzed in this whitepaper from each of six functional areas.
The three Product Risk functional areas are:
- Code Quality: Is the code good enough to be expanded? How much investment will be required to clean up technical debt?
- Development Process: How disciplined is the software development activity, and is the observed variation expected and desired?
- Team: Who are the code’s subject matter expert developers, and do they still work at the organization?
The three Product Risk functional areas are:
- Open Source: Might the code be legally required to be shared for free due to what third party code has been used?
- Code Security: Are there risks to the code or client information being hacked?
- Cyber Security: Have code or user credentials been hacked or accidentally shared? Is email set up to prevent phishing and spoofing?
The six metrics analyzed here are a subset of all metrics collected, however each of the metrics included are among the 12 most important identified by Sema’s research. [marco, link to the blog: https://www.semasoftware.com/blog/twelve-key-cto-and-cio-metrics-of-codebase-health
Findings Part 1- Product Risk
Metric #1 – Core Technical Debt (Functional area = Code Quality)
Definition
The Core Technical Debt Ratio is a rollup of four “imperfections” in the code common across languages: (1) insufficient unit testing, (2) duplicate blocks of code, (3) excessive complexity, and the presence of line level warnings (like what Grammarly finds in the English language).
We use “imperfections” in quotes because all code does, and should, contain technical debt suitable to its size, stage and industry. However too much technical tech impedes developer effectiveness and can lead to bugs and defects.
The core technical debt ratio compares the estimated amount of developer time to remediate the issues divided by the number of lines of code. As such, it is a size-adjusted comparison of code quality across organizations.
Findings
Youngest companies have the lowest average amounts of core technical debt, adjusted for lines of code.
Mid-stage companies have the highest average amounts.
Implications for Ongoing Operations
Companies should make technical reduction investment decisions in line with the company size and stage.
Specifically, earliest stage companies should are working on product market fit, and not prioritize technical debt reduction.
Mid-stage companies typically have the financial resources to reduce technical debt.
Later stage companies may have legacy products where tech debt reduction does not pay off.
Implications for M&A
Technical due diligence standards for code quality need to take into account the organization’s size and stage.
Metric #2 – Average Developer Activity (Functional area = Process)
Definition
Average Developer Activity is the number of commits carried out by developers over a given period of time. Consistent or increasing average developer activity increases the likelihood of meeting product roadmap goals.
Declining or varying developer activity could be warranted – for example, due to vacations or for EdTech companies with seasonal release cycles – but the variation should be understood and predictable.
Findings
Developers at 2-5 year old organizations commit the most frequently on average, twice as frequent as the oldest companies.
Within segments, there is a 200-400% variation in average developer activity.
Implications for Ongoing Operations
The hypothesized explanation of these results is that developers at more established companies are carrying out more intricate work such as bug fixes requiring more thinking and relatively less committing, compared to newer companies creating new features and functionality.
This would imply that measures of developer activity must be treated with significant caution, and curiosity rather than judgment, to understand the underlying factors and assess whether mathematically estimated developer activity is in line with business and technology objectives.
Implications for M&A
During a deal, developer productivity should be compared to similar-stage companies.
Post-close, there is opportunity to learn from similar companies, given the variation observed within segments. For example, many Private Equity-backed software and technology-enabled companies demonstrate best practices here in terms of knowledge sharing.
Metric #3 – Developer Retention (Functional area = Team)
Definition. Developer Retention is the percentage developers who’ve written a meaningful portion of the code who are still contributing code.
Sema’s qualitative research indicates that developer retention is the most important single indicator of codebase health, and the future ability to developer the roadmap with predictability, for two reasons.
One, developers keep the mental model of the codebase in their heads, despite best efforts towards codification, and losing that expertise slows down delivery.
Two, as leading DevOps expert Dave Mangot has noted in his recent book, low developer retention is an “anti-pattern” that can indicate other practices that impede developer effectiveness, as developers choose to leave suboptimal environments.
Findings
Developer Retention of subject matter experts is highest at companies 6-10 years old, lowest at companies less than 5 years old.
A 200-600% variation is observed within segments, significantly greater than the variation observed across company ages.
Implications for Ongoing Operations
These results are surprising. The hypothesis is that younger companies experience higher turnover due to loss of funding and developers “voting with their feet” about less experienced managers.
Given the criticality of developer retention to product roadmap goals, identifying causes of lower retention and remediating them should be a top business priority.
Implications for M&A
Developer retention of subject matter experts is a key M&A risk, especially for younger companies, and should be included
Findings Part 2- Compliance Risk
Metric #4 – Open Source Legal Risk from Referenced vs. Copied Code (Functional area = Open Source)
Definition
All Open Source code, and all code, comes with licenses. Some are permissive and do not present legal risk. Other licenses, such as “Copyleft” licenses, generate legal risk for the business if used incorrectly, specifically it generates a legal obligation that the organization’s code be shared for free.
It is generally preferred to reference Open Source code by using package managers, this organizes the code and facilitates compliance reviews and maintenance. What are third party code package managers
However, 100% of companies analyzed also have at least some Open Source code directly copied into their repositories. This is otherwise known as “In-File” third-party code, distinct from “Referenced.”
The metric analyzed here is the ratio of high-risk Open Source licenses copied into the code to high-risk licenses in referenced code.
Findings
There are 77 times more at-risk licenses from copied third party code, copied into the code, than known, referenced code.
Older companies have a smaller gap than younger companies.
Implications for Ongoing Operations
Companies need to invest in tooling to comprehensively identity Open Source legal risk given the prevalence of hidden, in-file risk.
Hypothesis is that older companies are investing in such tooling already.
Implications for M&A
Open Source legal review / Software Bill of Materials can’t rely on self-reporting during diligence, as companies are likely not aware of the full scope of the risks involved.
Metric #5 – Ratio of In-File third party security warnings to third-party security warnings (Functional area = Code Security)
Definition
In-File security warnings are vulnerabilities within an organization’s code inside its version control system. There are tens of thousands of such vulnerabilities, including cross-site scripting and hard coded passwords.
Third-party security warnings are vulnerabilities found in externally referenced code, such as Open Source code. A research institution maintains a list of such vulnerabilities (Common Vulnerabilities and Exposures, or CVEs) to share among practitioners and researchers for remediation and prevention.
The metric analyzed here is the ratio of high-risk In-File security warnings to high-risk third-party security warnings.
Findings
There were fifteen times more security warnings from code that developers wrote than from third-party code.
Mid-stage companies have the largest gap.
Implications for Ongoing Operations
Hypothesis is that middle-staged companies are focused on growing fast after product market fit so security warnings are a lower priority. Too, later stage companies are investing in tooling to identify these risks.
Companies need to invest in developer tooling to identify these risks, and Product/Sales/Finance needs to invest roadmap time in stage and industry-appropriate security remediation.
Implications for M&A
Security risk standards need to be adjusted for the stage-specific standards.
Security remediation is an important post-close activity for almost every company.
Metric #6 – Number of Cyber Security Risk areas (Functional area = Cyber Security)
Definition
Cyber security risk refers to potential opportunities for intentional or accidental information loss and system breaches detectable from a review external to the codebase. This includes personal information about organization’s employees available in dark web databases to hackers as well as the inadvertent sharing of sensitive or confidential information in the Open Source community. As such, Cyber Security risk can be considered “external” to the codebase, while code security is “internal.”
This metric counts the number of cyber security risk areas among 13 detected through Sema’s scan.
Findings
All companies scanned have risk in at least 3 of 13 risk areas.
99% of companies scanned have incomplete email setup of security standards (e.g. DMARC, BIMI).
Implications for Ongoing Operations
Companies need to invest in cyber security tooling covering a range of critical risk vectors, and also in remediating the findings.
In particular, incomplete email setup is both a security risk and negatively impacts email deliverability. For the security of the business and the success of sales and other externally-facing organizations, complete email setup should be a near term priority.
Implications for M&A
The prevalence of cyber security risks suggests that cyber security should be an operational risk, not deal risk, and a priority for post-close activity.
Conclusion and Future Research Agenda
Future Sema Research whitepapers will cover at least five areas:
Cross-Industry comparisons, such as, how do the profiles of tech-enabled and software companies differ?
Intra-Industry comparisons, such as, how much variation is there between Healthtech companies?
Cross-Functional comparisons, such as, do companies with more Compliance risk have more Product risk?
Product-level comparisons, such as, which of these trends hold up when comparing individual products within organizations?
New segmentation, such as, is there a different grouping of companies with better explanatory power?
For More Information
These metrics are included in the 12 metrics core to codebase health, you can read more here.
A free, anonymous calculator is available to calculate an organization's codebase health score using the 12 metrics-- link here.
