The Real Danger May Be AI as Dumb as Us

U.S. President Donald Trump’s rechristening of artificial intelligence as “super intelligence” seems unlikely to stick, executive order notwithstanding. But it was at least good for some ironic smirks among the very same AI “doomers” and effective altruists that the administration has spent the past few weeks bashing. If you’ve been following AI debates for a while, “superintelligence,”—albeit spelled as one word, not two—immediately brings to mind the 2014 book of that name by the Swedish philosopher Nick Bostrom, a foundational text for those who see AI as an existential risk to humanity. U.S. President Donald Trump’s rechristening of artificial intelligence as “super intelligence” seems unlikely to stick, executive order notwithstanding. But it was at least good for some ironic smirks among the very same AI “doomers” and effective altruists that the administration has spent the past few weeks bashing. If you’ve been following AI debates for a while, “superintelligence,”—albeit spelled as one word, not two—immediately brings to mind the 2014 book of that name by the Swedish philosopher Nick Bostrom, a foundational text for those who see AI as an existential risk to humanity. But the doomer vs. accelerationist debate—over whether super-intelligent AI, or “artificial general intelligence,” will usher in either an age of unfathomable prosperity or the literal apocalypse—sidesteps an uncomfortable reality. AI is already capable of causing a global catastrophe, not because it’s an all-powerful machine god that sees puny humans as a hindrance to its goals, or is misaligned with human values to the point that it turns us all into paperclips, or even is used to design a new bioweapon. It’s dangerous simply because people increasingly trust it even though it still screws up all the time. Some of the most concerning security risks from AI may come not from super intelligence but from what researchers sometimes call “jagged intelligence,” the way in which models can appear to be vastly more knowledgeable than humans in many respects but still lag bafflingly behind what we’d call common sense. When we put trust in those systems anyway, disasters can happen. AI targeting systems have likely already been involved in mistakes resulting in civilian casualties in Gaza and Iran. In September, reports emerged that a hallucinating AI had helped generate an intelligence report that reportedly nearly led to the U.S. military boarding a Chinese ship, potentially sparking a confrontation between two nuclear-armed superpowers. None of this has stopped the United States and China from aggressively pushing to integrate AI into military systems, including nuclear command and control. Nor have civilian errors such as the mass deletion of firms’ entire databases slowed these models’ rapid adoption in the corporate world. Some of the closest-call nuclear catastrophes of the Cold War were caused by malfunctioning technology, ranging from a defective 46-cent computer chip in a NORAD computer to an early-warning system misinterpreting sunlight bouncing off clouds as a missile launch. Overreliance on automated systems led to the U.S. military accidentally shooting down a civilian Iranian airliner in the 1980s and its own jets during the Iraq War. Today’s AI-enabled systems are more advanced, but that very sophistication—combined with how opaque their decision-making processes can be—can also lead human users to be less likely to exercise oversight or question their conclusions, a phenomenon that researchers call automation bias. Their jagged intelligence can mean that enormous sophistication goes along with elementary errors. The limits and possibilities of AI’s jagged intelligence were demonstrated by this year’s hacking of U.S. tech company Hugging Face by an OpenAI model that escaped its testing environment. By all accounts, this wasn’t self-sovereign intelligence, but reward-seeking behavior by a model that proceeded to do something that its designers clearly did not want it to do. In a national security context, the stakes would be far higher, said Jack Shanahan, a retired Air Force general who served as the first director of the U.S. Defense Department’s Joint Artificial Intelligence Center. “Imagine the Hugging Face thing, but it’s more like Stuxnet,” Shanahan told me, referring to the computer worm that caused significant damage to Iran’s nuclear infrastructure in 2010. “So we end up breaking into the Chinese nuclear command-and-control systems without ever intending to.” Jagged intelligence can also inform how AI models “think” about high-stakes situations. When researchers have leading large language models play war games simulating nuclear standoffs, they are nearly always more aggressive and escalatory than their human counterparts; far quicker to resort to nuclear threats and signaling. It’s not clear whether this is something inherent in their decision-making or if they’re simply trained on decades of Cold War-era deterrence theory and applying it more dogmatically than a human in a real-world situation would. The escalatory dynamics could only be compounded if, instead of responding to human opponents, the AI models are responding to each other. In his book Army of None, former Pentagon official and defense theorist Paul Scharre draws an analogy to the 2010 “flash crash,” during which the Dow Jones industrial average lost nearly 10 percent of its value and then recovered within half an hour, a bizarre incident caused by the cascading responses of trading algorithms set to respond in the blink of an eye to changing market conditions. The concern, in a military context, is that AI early-warning systems and automated weapons could respond to each other, escalating a conflict faster than humans can manage it. (Notably, this is also a Cold War-era concern: As far back as 1961, Norbert Wiener, the pioneer of cybernetics, warned of a “push-button war” resulting from machines making decisions faster than humans can reverse them.) It’s not only military AI that has potentially dangerous implications in a crisis. Rebecca Hersman, the former director of the U.S. Defense Threat Reduction Agency, has argued that increasingly sophisticated and difficult-to-detect disinformation could cause crises to develop in more unpredictable ways, a phenomenon that she terms “wormhole escalation.” Consider the 2022 deepfake video depicting Ukrainian President Volodymyr Zelensky ordering his troops to surrender, or the 2023 faked image depicting the Pentagon on fire that was picked up by Russian state media outlets and briefly caused a dip in the stock market when it was spread widely on social media. During the four-day military conflict between India and Pakistan in 2025, misinformation and deepfakes, much of it generated by AI, spread rapidly in both social and traditional media on both sides, including a report, backed up by a doctored document, claiming that India had attacked a Pakistani nuclear power plant, causing radiation leakage. Generating fake images and articles might be AI’s least glamorous function, but in certain circumstances, it could be just as dangerous. Some 99 percent of AI “slop”—the mass-generated fake images that now spam social media—may be ignored; that remaining 1 percent could spark catastrophe. While it’s not impossible that future highly advanced, self-sovereign AI systems could turn on humans or even convince them to turn on each other, far more likely in the short term is that hallucinations, poor design, or misinformation will lead to a disaster. Since the dawn of the nuclear age, the risk that human civilization would destroy itself through miscalculation, false confidence, or simple stupidity has always been greater than the risk of it happening deliberately. At this moment, the risk comes not from the fact that artificial intelligence is smarter than us, but from the fact that in some instances, it might be just as dumb.