Three scary AI security mistakes haunting enterprises

The enterprise AI nightmare is not a killer robot, but an erosion of our ability to see and control what’s running in our own environments. It’s time to wake up. There is a lot of talk about the coming enterprise AI reality, in which AI finally arrives in production systems. You might not know it, but this reality—or nightmare, depending on how you handle it—is already happening. It all starts with a “pilot,” a “prototype,” or a “side project.” Maybe someone builds an internal dashboard with an agent. The dashboard quickly becomes indispensable, and all of a sudden the experiment becomes production. Along the way, no one thought to ask the boring, inconvenient questions: What exactly was pulled from npm, PyPI, or Docker Hub? How is (or was) authentication configured? Is anyone watching for supply chain attacks against the tools and libraries the agents chose? And it’s not just a one-off project here or a couple of applications there. AI is enabling organizations to generate more code and ship more products and projects, more quickly, than ever before. By the time security teams get a look, the business is hooked and there’s no turning back. An actual nightmare has begun. There are three major problems that make the nightmare real. Components you never explicitly chose When you ask an AI agent to build an app, it doesn’t just spit out a single script. It quietly assembles an entire ecosystem around whatever problem you’ve described to it. It pulls in a web framework, grabs a bunch of libraries, stands up databases, and then it potentially builds everything on dependencies in container images. From a productivity perspective, this is awesome. However, from a security standpoint, it’s worrisome, to say the least. When I’ve built apps like this myself, I couldn’t begin to tell you all of the components that were being used unless I went back and asked the agent to explain itself. We live in a world where anyone can publish to npm or PyPI, and we’ve seen attackers slip malicious packages into those ecosystems or compromise ones that are widely used. Some of the recent incidents have involved security and devops tools themselves pulling a compromised dependency, running it as part of CI/CD with elevated privileges, and quietly exfiltrating secrets or tampering with builds. I personally experienced this type of compromise a couple of months ago, and had to update all of my credentials in GitHub. Pulling unvetted code is bad; now layer AI agents on top of that. They default to whatever is easiest to discover and integrate. If a package solves a problem in front of the agent, the agent will add it. This is the old “download a random library from the Internet” problem, but now it’s on autopilot, at scale, and moving at a pace we’ve never seen before. To solve this problem, we must provide the agents with an innate sense of our risk tolerance, an approved components list, our desires around logging, etc. We can do this with spec files and what the industry calls constitutions. Collectively, this is called harness engineering, which we will talk more about later. Skills shifting from code to architecture There has been a lot of hand-wringing about whether junior developers will ever really learn to code if AI is doing all of their coding for them. That’s not what worries me. I think it’s fine to let an agent spit out code. It’s a job they are really good at. What they are not really good at is identifying and avoiding problems in code. I haven’t written code in quite some time. I can, but it doesn’t make sense for me to do so. What is worth my while is noticing when an agent suggests something dumb or even dangerous (or both). For example, while working on a recent personal project, an agent proposed exposing a memory server on the public Internet with no authentication. The agent wired things up so smoothly that, at first glance, everything looked fine and just worked. But then I paused and asked, “Wait, how is this actually authenticating? Where’s the password, secret token, or OAuth in this flow?” Turns out it wasn’t authenticating and there was no password. If I hadn’t taken that beat—and then argued with the agent for a while—the app would have gone live with no protection. So, the skills issue isn’t about whether we will lose the ability to code but rather whether we have the ability to ask questions and be discerning, and whether we have the understanding to know when something doesn’t look or even feel right. Do organizations have people who know what a dangerous software pattern looks like when the agent suggests it? You need people who can recognize when an authentication flow is too permissive, when a data store should never be exposed beyond a certain boundary, and when an architecture has become such a steaming pile of technical debt that the right answer is to throw away a whole layer and rebuild it. You need people who know that “what works” isn’t the same as “what’s safe” or “what’s right” and who can argue back with the agent when the former doesn’t line up with the latter. It’s not about syntax. It’s about architecture, supply chain awareness, and the willingness to say, “We’re tearing this down and doing it right,” even when the prototype looks good on the surface. Teach your AI-assisted coders basic security principles, basic architectural patterns. The AI will teach them the more advanced stuff, as long as they keep asking questions. Agents with no harness The third problem is that we’ve unleashed some very capable agents into our development workflows without treating them like first-class actors that need governance. Many organizations are wiring AI assistants into a repo or IDE and letting them scaffold projects and pipelines. Maybe they bolt on a security scanner and declare “AI enablement.” That’s not a governance model, that’s optimism (and not even cautious optimism). Indeed, a code-generating agent with broad access to your repos, your CI/CD pipeline, and your artifact registries is effectively a hyper-productive and not-very-well-trained junior developer with access to the Internet and no ingrained sense of organizational policies. It can introduce new tools, new dependencies, and new patterns faster than your review processes can handle. In my personal projects, I’ve started to think of this as what AI coders call a harness-engineering problem. For every agent that’s responsible for building or wiring code, I try to put other agents in the loop that are responsible for tearing it down, at least conceptually. For example, one agent focuses on security and looks for obvious vulnerabilities and bad practices. Another looks at architecture and points out when the app design is veering into unmaintainable territory. A third looks at performance and reliability issues, which are themselves a kind of security concern when you think about things like denial of service and resource exhaustion. Pair this with constitutions that give the agents first principles on architecture, security, and design, and this is no longer vibe coding, it’s harness engineering at scale for all of your projects. What I am doing isn’t perfect; there is no perfect in this space, because these are non-deterministic, statistical tools. But, many organizations aren’t even doing this. In effect, their agents are freelancing. They’re vibe coding. They’re not constrained to trusted registries or hardened base images. They’re not required to log their decisions in a way that security can audit. No one owns the harness, and that means a lot of implementation decisions have fully shifted from humans to systems that no one is really watching. New problems require new thinking The enterprise AI nightmare is not a killer robot; it’s the erosion of our ability to see and control what’s running in our own environments at the exact moment our velocity is exploding. The danger is in ceding your agency. It’s in shipping applications that internal and external customers love—and don’t want to give up—but inherently aren’t safe. Right now, someone in your organization is using AI to build a capable app, pulling in who knows what from who knows where and adding it to your infrastructure. The good news is that these problems are identifiable. They are also solvable, although it will take a new form of thinking than what solved problems in the past. You must think statistically, and declare constitutions with first principles. You can standardize trusted stacks and registries. You can retrain people around architectural security rather than just “secure coding.” You can start treating agent harnesses as systems that deserve design reviews and edits. But, none of that can happen until the enterprise is willing to admit that the nightmare is already here.