Table of contents
For enterprise IT operations that are in a never-ending battle against budget cuts, the possibilities of coders using GenerativeAI (GenAI) tools presents a tempting opportunity.
On the one hand, the technology promises massive productivity and efficiency benefits for programmers, to the point where it could allow a couple of programmers the ability to perform the work of dozens. On the flip side, GenAI coding fails frequently and differently, which means that between hallucinations and completely fictitious libraries, the technology could deliver security holes, compliance nightmares, pen-testing challenges and even scuttled acquisitions.
The same generative AI tools that are supercharging the work of both skilled and novice coders can also produce flawed, potentially highly dangerous code.
The reality is that the potential efficiency benefits are so great that few C-levels can resist. But are they deliberately underplaying the dangers? Faustian bargain indeed.
GenAI coding is essentially the third chasm in the history of coding. The first leap was in the 1940s, when assembly language made the move from hand-written to computer coding. It was painful and slow, but ultimately much faster than manual.
The shift from coding by hand to programming on computer had serious consequences. The carefulness was substantially different, resulting in far more errors. It’s a familiar pattern: as convenience increases, accuracy decreases. When writing code manually, programmers needed to slow down.
The second shift was Open Source coding, which started in the 1950s (when almost all software was free and shared), got a boost in the 1980s from the GNU Project and the Free Software Foundation, and accelerated in the 1990s with Linux, Apache and Netscape. The sharing of code snippets accelerated much of the more tedious work, especially when the program needed functionality that was already widely used. That ability to repurpose existing code was a grand leap in coding efficiency, but that also exposed those programmers to backdoors, holes due to sloppiness and then holes due to maliciousness. But it was a massive leap in efficiency, accompanied by a massive leap in risk.
Now we come to the third phase, the soaring use of GenAI for software development. Just like Open Source, those efficiency boosts came with a cost.
Those risks–and benefits–are both limited while GenAI is being used lightly, or as a programming assistant that offers periodic help to a human coder. But as more companies fall in love with the efficiencies and use it more, the risks will grow. Hallucinations don’t happen with every line of code written. It’s random. That means that as the number of lines of code written increases exponentially, so will the hallucinations.
The more that companies consider relying on GenAI alone for coding, with human coders reassigned to strategy and functionality decisions, the more devastating the potential risks.
Given the significant benefits to both coders and organizations seeking to increase their development velocity, we can confidently predict that GenAI coding tools adoption will soar in the Fortune 1,000.
And frankly, we wouldn’t want to fight this trend even if one could (One couldn’t). Go back to the Open Source example—ask any development team or software engineer if they are net better off as a result of Open Source. Younger developers might look at you askance – Open Source frees up engineers time to work on what only they can do—not creating a web framework from scratch (why re-invent the wheel) but solve new problems based on the specific context of their organization.
The only rational move is to put in place safeguards. If you can’t prevent the problems, then you need to closely monitor and test to quickly identify and try and mitigate the problems.
Let’s break down the kinds of coding risks that are most likely:
- Simple mistakes, typos, code conflicts.
- Hallucination problems such as referring to other code that doesn’t exist.
- Backdoors coded because the LLM wants to be able to return at a later point and make repairs. (The problem is that if the LLM can later gain backdoor access, so can cyberthieves.)
- Functionality issues.
- Lower value Intellectual Property, both because GenAI means that other development teams can build competitor products faster, and also that certain IP protections, such as copyright and trade secrets, are going to have to change. Legally, when does an AI tool helping a coder morph into AI becoming the author?
When working on ways to mitigate the generative AI coding risks, it is important to keep the five categories above in mind. That is because each problem category is best addressed by very different countermeasures.
For example, functionality deficits are handled quite well by most application testing procedures. Making sure that an application, especially a homegrown application, does exactly what it is supposed to do is the first thing that is generally checked. Contrast that with hallucination issues which are almost impossible for traditional software checking mechanisms to discover.
Consider the fact that the majority of software coding problems are exacerbated by GenAI code but are not unique to GenAI code.
Coding sloppiness is bad, regardless of whether it’s because of human corner-cutting/shortcuts/laziness or an algorithm that–please forgive the anthropomorphism–lets its mind wander.
The consequences are more significant than the code failing. When it introduces cybersecurity holes and other vulnerabilities, in addition to more compliance problems, your business could see litigation from customers hurt in a data breach.
The nightmare that few IT executives think about are M&A disasters. Investors and potential acquirers are going to be seriously examining your code. Coding problems won’t merely make them reconsider their acquisition interest, but coding glitches are going to make potential acquirers wonder about what other sloppiness exists in your operations? Are the books accurate? Are you properly supporting your customers?
Then there are the legal issues, intellectual property and copyright. Do your people truly know where GenAI is coming up with its suggestions? How much transparency does your team actually have into the algorithm’s training? Do they know for certain where data goes after it’s entered?
For right now, you don’t necessarily need to have exact answers to all of that. But you absolutely do need to have a concrete plan for not only getting those answers, but for constantly getting updated answers.
So many of those answers can be discovered by a sophisticated analysis of your coding efforts. But like everything else with GenAI, it’s not static. GenAI is going to morph and evolve, which means your code-checking strategy needs to do the same.
