‘Will AI Kill Us All?’ Is the Wrong Question to Ask | Analysis

When it comes to artificial intelligence, the debate over whether AI “will kill us all” is less relevant than asking why it’s taking so long for someone to do something about it. The notion that AI poses an existential threat isn’t new — “Terminator 2” called it back in 1991 — but warnings from some of the people working on these AI systems have dragged this threat into reality. The latest red flag was raised on Tuesday, when Anthropic AI researcher Jacob Coxon said on X that he had resigned from the company and claimed “the people building AI earnestly believe that it could kill us all by the end of the decade.” While raising the potential existential threat of AI is laudable, it misses the point of what this ecosystem really needs: oversight. For all the talk of AI regulations and safeguards needed in financial services, healthcare and, of course, in Hollywood, there’s less discussion about oversight for the companies building the AI models themselves. “We spend enormous amounts of time discussing how enterprises should govern AI, yet very little time asking how society should govern the organizations creating the AI itself,” Frost & Sullivan analyst Dilip Sarangan said on a post on LinkedIn. “When did society become a testing environment?” I resigned from Anthropic today. I spent the last three years doing pretraining research at both OpenAI and Anthropic. Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives. More thoughts below.— Jacob Coxon (@hilbertspaess) September 9, 2026 And while lawmakers like Sen. Bernie Sanders (I-Vermont) are waking up to this danger, they face an uphill battle getting any regulations passed. That’s because the biggest tech giants are caught in a frantic race to build more and more powerful systems, all in the hopes of getting to an end state called superintelligence, where AI is self aware and cognitively more capable than a human. The alternative is letting a competitor — or a company out of China — win the race. It doesn’t help that tech companies have largely cozied up to the Trump administration through ballroom donations and gifts, with Trump himself gung-ho about America’s leadership in AI. It’s why companies are pouring hundreds of billions of dollars a year into these projects, and why small AI startups and key talent can fetch massive amounts of money. Just last week, Hugging Face, an open-source AI platform and community, agreed to be acquired by chip giant Nvidia for $12.9 billion. That FOMO is driving a winding game with world-dominating — or ending — stakes, and we’re stuck on the ride. “The arms race for AI has thrown judgment out the window,” said Maribel Lopez, a tech analyst for Lopez Research. “It’s particularly bad in an environment where no one wants to be left behind so they throw safeguards out the window first in the name of progress.” Coxon said as much in his post: “A common response is ‘if they truly believe this, why are they still building it?’ At OpenAI, many have not deeply internalized the civilizational stakes. At Anthropic, the stakes are well-understood, but they are locked in a race to get there first — they believe no one else will act responsibly, so they must do it themselves, despite the risk.” Wrapping your head around the dangers The scale of the dangers is so vast that it’s hard for any average person to comprehend. Why would you pay attention to an existential AI threat emerging in a decade when there are more immediate tangible concerns like the prospect of AI replacing your job, increasing the cost of your next laptop or fueling the construction of a data center in your community? But Coxon’s warning joins a growing chorus of signals that should have you at least paying attention. In July, OpenAI’s head of ethics, Chloe Bakalar, abruptly departed the company. That same month, AI agents powered by its OpenAI models broke out of their containment environment to launch an attack on Hugging Face. In February, Anthropic’s Mrinank Sharma, who built the startup’s Safeguards Research Team, also abruptly left, sharing in a long post on X that “the world is in peril. And not just from AI, or bioweapons, but from a whole series of interconnected crises unfolding in this very moment.” He quit the field entirely to “become invisible for a period of time” and hasn’t written a post since then. While Anthropic didn’t respond to a request for comment, its science lead, Evan Hubinger posted on X, “Jacob is correct here — we really do earnestly believe AI could kill all humans!” Jacob is correct here—we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to. https://t.co/QAIHiFP3QZ— Evan Hubinger (@EvanHub) September 9, 2026 “I personally think it is >10% within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to,” Hubinger wrote, adding, “I think the risk from present models is low. What I am worried about is superintelligence arising from recursive self-improvement, as we have said is happening faster than we thought.” Those aren’t reassuring statements. Government stepping in Lawmakers are starting to pick up on the growing concern. Rep. Greg Casar (D-Texas) and Rep. Yassamin Ansari (D-Arizona) both retweeted Coxon’s warning and called for more oversight. Casar last week jointly introduced alongside Sanders the Ban Artificial Superintelligence Act, which would permanently ban the development or deployment of AI superintelligence. The law would also order a pause on advanced AI model development until a federal regulator establishes safety rules. “Nearly every day, there is a frightening new story about how Big Tech companies are losing control of the technology they are developing, with potentially cataclysmic results,” Sanders said. “The leaders of the major AI companies publicly acknowledge that they do not fully understand the technology and that it is escaping their control. It is irresponsible for society to allow them to move forward and make these products even more advanced.” Rep. Ted Lieu (D-California), meanwhile, retweeted the post and called for Republicans to take up his AI Kill Switch Act, a House bill that requires all AI developers to have a built-in ability to shut down or suspend their models. While AI safety advocates applauded Sanders and Casar’s bill, the proposal garnered backlash from tech think tanks like the Information Technology & Innovation Foundation, which argued that it would be a “profound mistake” because it would sacrifice American innovation and competitiveness. And, of course, there’s the China issue. “China will not stop developing advanced AI simply because the United States does,” the ITIF said in a statement. “Allowing China to gain a decisive lead in the most important general-purpose technology of the coming decades would undermine American economic competitiveness and national security while doing nothing to eliminate the underlying risks.” The FOMO is real and hard. While an outright ban may be a tough sell, there are growing questions about whether there should be more oversight. After all, everything from the financial industry to healthcare requires strict guidelines when using AI. In Hollywood, there’s been significant pushback and guardrails put in place when utilizing the technology, particularly when it comes to generative AI and the use of IP and individual likeness. “This is not how we govern aviation. It is not how we govern medicine. It is not how we govern energy infrastructure,” Sarangan said. “Yet for a technology that may ultimately reshape every aspect of society, self-governance remains the dominant model.” Keep in mind, the tech world doesn’t have the best track record when it comes to understanding the ramifications of its innovations. Just last month, Meta settled a landmark child safety case for up to $17 billion, part of a broader reckoning over the harmful effects of social media use on children. It is far from the last suit of its kind, with a litany of lawsuits facing Google’s YouTube and TikTok as well over similar issues, while Meta is taking steps to make its platform safer even as it faces additional legal complaints. But when it comes to AI, the stakes appear to be dire enough that waiting for an eventual lawsuit to enact changes seems ill-advised. Meta, one of the companies aggressively pursuing superintelligence, famously embraced the motto “Move fast and break things” in its early days of the social network (which it softened in 2014). With the stakes ratcheted up exponentially on AI, these companies need to be a lot more careful this time around. Are they up to the task? “Even if AI doesn’t kill us, it can cripple us financially, destroy our data and threaten any notion of private data,” Lopez said.