In December, AWS suffered an outage that took services offline for 13 hours. It has since been deduced that the incident was reportedly caused by an AI agent, Kiro, autonomously deciding to “delete and then recreate” a section of the AWS environment. Whilst we’re yet to see other major tech companies suffering a similar fate, Amazon could be remembered as ‘patient zero’ for major AI coding errors.
Gartner predicts that almost all software engineers (90%) will be using AI by 2028 – up from fewer than 14% in 2024. This prevalence means that the AWS outage could have happened to any company, and it certainly won’t be the last organisation to have operations disrupted by machine-made coding errors. The incident is indicative of a wider problem that will only become amplified as human engineers are replaced by AI-driven ones.
AI certainly helps to streamline development processes, but it’s clear that even well-resourced organisations are allowing AI to create or make changes to critical systems unchecked. By now, there’ll be billions of lines of AI-generated code running in production around the world. While AWS’s AI coding error triggered an outage, many other AI-generated lines operating in the wild are likely to have introduced vulnerabilities into software development pipelines.
Once in production, finding these vulnerabilities and correcting them is like looking for a specific needle in a haystack-sized pile of needles. Organisations have little way of tracking who or what wrote code, for what purpose and when. Although we’re yet to see a major event where these AI-created gaps have been exploited, history shows us that cybercriminals will be relentlessly probing to discover any weak point in defences.
And whether it is fair or not, security professionals will feel the sharp end of criticism for AI coding failures that lead to security incidents. So, it is up to us to take the lead on building resilience and setting security standards now, before it is too late.
Although AI-driven coding errors are a new area of vulnerability, many existing defence mechanisms still apply. As with traditional software build pipelines, organisations must ensure security is built into every phase of the build process to reduce the chance of vulnerabilities being baked into software. Security cannot be an afterthought that is addressed at the end of a project, regardless of whether it’s AI or a human writing code. Emphasising this point to engineering teams is vital.
The profession must also create and enforce policies around safe, effective use of AI. Rigorous testing and ensuring that any code written by AI is evaluated by an experienced professional must be top of the agenda. Technology can help to enforce these policies – for example by limiting access to AI coding tools – but colleagues will also need to be educated around why these strategies have been put in place.
Broader education of colleagues beyond developers over the risks of AI coding, the vulnerabilities that cybercriminals might exploit, and best practice will also be crucial. AI has lowered the bar for entry to coding, meaning almost any employee – regardless of their experience or skill level – can get AI to build software using just a few simple prompts. This means that all employees must be made aware of the risks involved with using AI to build software.
Finally, there are technological guardrails that can be deployed to help drive standards and resilience. Software Bill of Materials (SBOM) generators can help keep track of who or what has made changes to code and when. Regular pentesting can provide an additional layer of scrutiny beyond internal code analyses. And traditional forms of security – such as post-exploitation detection, firewalls and runtime controls – can help to discover, prevent and limit attacks.
As ever, new innovations create new threats. But through a mixture of people, process and technology, the security profession can instil best practice and drive resilience, helping to prevent cybercriminals from taking advantage of AI-written coding errors.