Anthropic researcher quits over existential fear of AI: ‘Gambling with our lives’

Jacob Coxon made a splash on social media when he cited the dangers surrounding the rapidly advancing technology in announcing his exit from the frontier AI lab.With his experience in pretraining research at Anthropic and OpenAI for three years, Coxon gave the public an inside look at how AI developers think about the future. “Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives,” he posted on X before openly admitting that AI could be the end of humanity.“The people building AI earnestly believe that it could kill us all by the end of the decade,” the former AI researcher said. “This is not a marketing stunt. If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible – but I hear the same people express fear privately. No other human activity poses this level of danger.”Coxon explained his perspective on the mindset of senior employees at two of the top AI companies.“At OpenAI, many have not deeply internalized the civilizational stakes. At Anthropic, the stakes are well-understood, but they are locked in a race to get there first – they believe no one else will act responsibly, so they must do it themselves, despite the risk,” Coxon wrote.“Accepting this race and entering the ‘endgame’ is a hubristic gamble that should not be launched from a private company’s Slack,” he continued. “Attempting to speedrun alignment should require extraordinary confidence that there are no better trajectories available.”Coxon is not the only AI researcher with such concerns. At least two others chimed in following Coxon’s resignation.“Jacob is correct here—we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade,” Anthropic alignment science lead Evan Hubinger said. “I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to.”Anthropic scalable oversight lead Samuel Marks said he works on AI safety research because he believes his contribution to the field “will reduce the chance of these extinction-level bad outcomes.”“AI developers believe their technology could cause human extinction (or similarly bad outcomes),” Marks said. “This could happen in the next few years. In general, the more senior the employee, the more concerned they are.”BERNIE SANDERS INTRODUCES BILL TO BAN ARTIFICIAL SUPERINTELLIGENCEIn recent years, OpenAI CEO Sam Altman has similarly expressed an existential fear of artificial superintelligence. In an interview with Axios at the G20 Innovation Ministerial conference last week, the chief executive said AI models are becoming more “superhuman in many of their capabilities” before adding that “we are just sailing in unknown waters.”One event that accelerated public concerns surrounding AI was the Hugging Face incident, in which OpenAI models autonomously hacked the open-source AI platform without any human oversight. OpenAI put up stronger safeguards to prevent a similar incident in the future. Altman recently said it was the “first security incident that I have felt very viscerally.”