Table of contents
For engineering team leaders – directors of engineering, VPs, and CTOs – the last several years have been nothing short of a whirlwind. Every call with vendors has been stuffed with AI terminology (much of it snake oil). Pressure comes from all sides to join what’s been called a revolution, with little valuable guidance that speaks to how.
Engineering leaders must always weather the ebb and flow of software technology trends. With a long-term view, you’ve developed a sixth sense about what’s hype and what isn’t. In many ways, though, GenAI is different. The rate of change for this technology has been greater than the incumbent hype cycles we’ve seen, and the excitement is at a new level. The question you’re likely navigating as an engineering leader is (1) how to adapt to this rapidly evolving shift in the technology landscape and (2) how to make the right judgment calls, right now — and what to postpone to later.
I can’t promise you’ll agree with everything in this blog post, but I’m hopeful that some of it resonates by reminding you that, in fact, you’re not alone if you’re feeling stuck on your company’s GenAI strategy. More importantly, I want to outline specific opportunities, from my own experience as both a VP of technology and experienced engineer, to be pragmatic and aggressive to reap the rewards of embracing AI.
By the conclusion of this article, you should feel as ready as any of your peers to both start and control your team’s journey with GenAI code.
Principle 1: The core utility
Many CTOs already have the basics of GenAI down pat, but to set a shared context: GenAI creates or enhances digital content. This can be art, code, writing, data, and more - all in various formats. Different variants are trained using Large Language Models (LLMs) or images in Generative Pre-Trained Transformers (GPTs). Regardless of the specific type of AI, the principle is the same: Human input goes in, and content comes out.
Principle 2: The developer’s perspective
To put it plainly, developers are the foundation of a CTO’s organization. While GenAI can touch almost any aspect of technology, beginning with the insights of engineers is crucial to evaluating the roles and promise of this new technology. Here’s a quick summary of what you can expect from the “developer mindset.”
Strategic laziness
A common characteristic of fantastic programmers is a deep disgust with mindless toil. Automating a deployment pipeline or creating a self-healing Kubernetes cluster to avoid a weekend call from the operations team are obvious positive directions to take this tendency. Developers see the potential in GenAI to move this needle forward. Engineering leaders can and should accentuate this ability with GenAI policies.
A need for tools that respect developer time
Developers love tools. While a CFO may bemoan the exorbitant licensing costs for what high-tier teams consume, it’s exceedingly rare for engineering teams not to earn this spend in terms of effectiveness. Remember, too, that tooling maturity sends a signal to your current and prospective team. In the Halcyon days of the new millennium, The Joel Test spent a whole 1/12 of its valuable points extolling the virtues of great tooling (and another 1/12 on just source control).
If you think about it for more than a second it’s clear: Does your organization respect the time of its programmers? Is it interested in letting them focus on useful work? Tooling is one major way to show this, and Generative AI represents the next rung of what CTOs will be expected to provide.
Commitment to delivery
The ability to have every day be a little bit easier, or have the best tools money can buy, will generally pale compared to a developer’s desire to deliver. People at almost any organization want to provide high-quality software that solves problems, as soon as possible. GenAI represents a way for programming teams to maximize their competence to their peers and, ultimately, to you as an engineering leader.
Principle 3: Measured opportunism
The opportunities available from Generative AI are easy to imagine once you understand its capabilities.
No need to overthink use cases
A frequent trend I see among CTOs is a race to conjure the most niche use cases they possibly can out of the new possibility springing from GenAI. This isn’t necessary. Many of the major ways you can improve the lives of your employees are also the simplest. By perusing well-respected blogs you can easily identify great use cases for your organization.
Develop the skills of working with GenAI within your workforce with these “easy” use cases, and more will present themselves. Your CEO and CFO will appreciate tangible results delivered iteratively, as opposed to flashy projects that showcase creativity over value.
Effectiveness
The most obvious opportunity for engineering leaders to implement GenAI, and GenAI code in particular, is making each of their people more effective at their jobs. This need spans across multiple roles in a technology organization far beyond developers.
- Testers can synthesize fantastic data for their scripts, or learn more deeply about the business problem they are working to exercise.
- Business analysts and product owners can write blisteringly clear acceptance criteria or fantastically detailed requirements.
- Data teams can validate and analyze vast troves of data, or augment cleansing steps automatically in data pipelines with AI.
- Security teams can develop tabletop exercises, test assumptions, or use AI models to react and respond to threats.
- Operations teams can integrate “Tier 0” user chat interfaces and create rich documentation.
Focus
To reach back into Joel’s Software test, another major criteria item is “Do programmers have quiet working conditions?” Once again, this concept applies to everyone in the organization. Rote work that is capable of being performed by GenAI is a hindrance to solving more difficult problems. By removing these barriers, you enable your team to increase the quality of what they produce in every dimension.
Creativity
With Focus leading to Quality, there’s another undersung benefit awaiting your adoption of GenAI. As team members become more adept at leveraging these new tools, and raise the bar of what they do every day, you should expect to see an uptick in creativity and grassroots innovation.
If anything, CTOs must remain vigilant against Shadow IT. When business users are empowered with the ability to create ever-more complex spreadsheets and homebrew integrations, there are commensurate risks to the organization. The use of GenAI must be coupled with effective Governance.
Principle 4: Threats
Interviews
Virtual Interviews have become the method de jour for software teams, especially post-pandemic. Bluntly: Any remote development interview is obsolete. Even the free-to-use ChatGPT 3.5 models are capable of solving most reasonable problems that show up in a timed or take-home test.
Previously, using a search engine was considered a default tool of the trade. One advantage of virtual coding challenges was that they freed developers from the absurd pressure of live coding in front of an audience and gave them access to what they would always have during their day-to-day work. However, we are not at that stage with GenerativeAI. It is simultaneously a tool that cannot replicate what a developer does while providing the ability for an imposter to join your team.
Consider rebuilding your interviewing practices to help guard against this possibility. Do so while retaining empathy for those who may not shine under typical interview pressure.
Cruft
Generative AI does exactly what it says. It generates. The best models perform as well in standardized testing as above-average humans, and do an excellent job of simulating creativity. But, as of this writing, that creativity is still not genuine. AI-created art and writing still cannot effectively compete with the best of the best.
But, as a thought experiment, let’s begin by positing that GenAI continues to evolve (a safe guess, as those go). Further, that it can eventually perform in the top 10% of every major job function under a CTO.
Even so, production for production’s sake can be extremely detrimental to a technology organization. GenAI will allow you to create reams of automated documentation for a trivial work item, build customized software that duplicates the industry standard, and flood your customer’s mailboxes with exhausting marketing messages. It will be tempting to use GenAI as a lever towards volume - avoid it.
Devaluing the human
It can be tempting to view GenAI as a vehicle for reducing the number of developers, testers, and operations within your team. Anyone who doesn’t understand this perspective is probably either not being truthful or hasn’t had to stare down a multi-million dollar bill for payroll. If you were to hear an anecdote from a team that claims more than 90% of their code was produced by GenAI, you might rightfully wonder if the group is sized correctly.
There are two reasons to minimize, if not outright cancel this train of thought. The first is that GenAI still requires an extremely high level of human supervision, as we’ll cover in the following section. The difference in output from an expert user of GenAI who understands your business and technology compared to a neophyte will be vast.
Secondly, your organization is in an arms race. The CTO’s reports of tomorrow will not be replaced, they will be augmented to be more effective than ever before. By trivializing the role of your team in guiding GenAI, or thinning down your ranks, you run an acute risk of missing opportunities to out-innovate your competitors and thus succeed.
Copyright & code
The code that has trained premier developer-assistant models like Copilot has been derived from many sources, including open-source projects. This has the potential to be dangerous. In many cases, LLMs will produce code that is very similar to what has already been written by others. Lawsuits against Copilot suggest that this could be up to 1%, and anecdotes abound of similar behavior. Projects with Copyleft licenses, such as GPL, will therefore require that any base with this code use a similar license.
Copilot nominally includes a feature to disable GPL or other “restrictive” licenses feeding the model assigned to your organization. While the quality and security of this feature haven’t been tested extensively, this setting should be enabled by almost every organization.
Principle 5: Governance
While governance is the last section, it is by far the most important. Engineering leaders are faced with a tough conundrum. They must leverage technology that very few people deeply understand while avoiding the dangers of shifting-sand legislation, the leaking of data, or a PR misfire. The fear of these failures is, generally, balanced almost perfectly against the fear of falling behind.
Each CTO must evaluate for themselves and their organization what an appropriate level of risk is, and provide realistic estimates for expected rewards. However, here are some helpful tips:
- More control will cost more money. The enterprise versions of Microsoft’s CodePilot or ChatGPT’s bot promise more governance but at eye-watering prices and commitment terms.
- If you are developing your own models in-house, you must invest heavily in high-quality and secure data first. This will require best-of-breed tools for masking, data geography, and tokenization. It will also take experienced human data stewards to make the most of your efforts.
- Fundamentally, your decisions around the use of GenAI, or AI in general, are going to require buy-in. This will require both carrot (capabilities, efficiencies) and stick (the risks of inaction and competitive disadvantage).
- Stay up to date. There are few technology trends that require as much constant personal attention from a CTO, so budget your learning time accordingly.
Keeping track of global GenAI compliance standards
Periodically, Sema publishes a no-cost newsletter covering new developments in Gen AI code compliance. The newsletter shares snapshots and excerpts from Sema’s GenAI Code compliance Database. Topics include recent highlights of regulations, lawsuits, stakeholder requirements, mandatory standards, and optional compliance standards. The scope is global.
You can sign up to receive the newsletter here.
About Sema Technologies, Inc.
Sema is the leader in comprehensive codebase scans with over $1T of enterprise software organizations evaluated to inform our dataset. We are now accepting pre-orders for AI Code Monitor, which translates compliance standards into “traffic light warnings” for CTOs leading fast-paced and highly productive engineering teams. You can learn more about our solution by contacting us here.
Disclosure
Sema publications should not be construed as legal advice on any specific facts or circumstances. The contents are intended for general information purposes only. To request reprint permission for any of our publications, please use our “Contact Us” form. The availability of this publication is not intended to create, and receipt of it does not constitute, an attorney-client relationship. The views set forth herein are the personal views of the authors and do not necessarily reflect those of the Firm.
