Explainer: Why is everyone talking about AI and the end of the world?

Explainer - The debate over artificial intelligence and how dangerous it is has reached warp speed in the past few days.The head of one of the world's biggest AI companies is calling for development to slow down as some warn the technology "could kill us all", while US President Donald Trump has dismissed anti-AI rhetoric as "sick".Many say the debate is getting increasingly urgent."I think it's the most important thing happening in the world right now," John Steidley, an AI researcher at Californian nonprofit Palisade Research, told RNZ's Morning Report.It's all moving incredibly fast, but here's a quick look at what has triggered the latest rounds of AI concern and what power brokers and politicians are saying.Anthropic CEO Dario Amodei has called for AI development to slow down.MICHAEL M. SANTIAGO/AFPWhy is everyone suddenly freaking out?Dario Amodei, the CEO of leading AI company Anthropic, posted an essay over the weekend urging developers to slow down and add safety standards.It's set off a firestorm of debate from the White House on down on a technology that is making headline news around the world, but is moving so fast that it's difficult for many people to even fully understand what it's doing."We must slow the pace at which we improve the capabilities of AI models," Amodei wrote, putting forward a three-step plan that would call for independent evaluators over safety, coordination between firms over their own standards and international cooperation to manage AI risks.Elon Musk and OpenAI CEO Sam Altman said they agreed with Amodei's concerns."The people building AI earnestly believe that it could kill us all by the end of the decade," said Jacob Coxon, who had worked at both OpenAI and Anthropic before he posted his warning on social media.That warning has echoed around the world in the past week."In AI, governments, companies and the rich (mainly) men behind them are chasing geopolitical, military and financial advantage by building it as fast as possible," the ABC's Alan Kohler wrote in an analysis piece published by RNZ.A backlash has been growing against AI's expansion, its effect on the environment and employment, as well as the lack of safeguards over the industry.US President Donald Trump, on the other hand, has taken a far more laissez-faire approach to AI regulation, claiming there is a "sick conspiracy" against the technology. He's said America must "win" the AI war against China and dismissed concerns about data centres."The only control or 'guardrails' that AI needs is a STRONG AND SMART (High IQ!) PRESIDENT, and the U.S.A. has that, in spades!" he wrote.OpenAI CEO Sam Altman.Matt Ramey / AFPWhat are some of the fears about AI?The two biggest fears about AI are around deliberate malicious deployment by humans or, more worryingly, autonomous replication by AI agents acting on their own initiative, Steidley said.A hacking incident in July by what was called "rogue" AI made news around the world.OpenAI said that one of its experimental models broke out of a testing space and hacked into an AI startup called Hugging Face.Connor Leahy, an AI expert and the US director of Control AI, told the ABC that the attack was "crazy"."The idea that something could just, without human oversight, escape from something like this and then autonomously decide to attack a different company is kind of unheard of."Steidley said that attack was a red flag for many in the AI field."I think one thing that we're very fortunate to have is this example of the Hugging Face attack from July, where AI got out of control and that gives us a chance to respond to that without any humans being severely harmed."There have been concerns about it being used to create new kinds of biological weapons or access nuclear weapons.There's also fears about the enormous amount of money being poured into AI and that it's creating a bubble in the stock market that will eventually burst, causing global financial chaos."I have a lot of hope that we can solve these problems, but we have to make sure that we don't move so fast that no human can keep up with what's happening," Steidley said.Is the panic overblown?That very much depends on who you ask.Timrit Gebru, an AI researcher and former Google employee, recently told Wired magazine that the talk of humanity's extinction at the hands of "machine-gods" is meant as a distraction from real issues it contributes to like climate change and job losses."Bosses are using AI as an excuse to get rid of their pesky workers," she told Wired.Internet big-picture writer and essayist Cory Doctorow has criticised the "AI-pocalypse" narratives and claims that AI is somehow "alive," calling LLMs (large language models) "a very sophisticated word-guessing programme" and accusing companies of fear-mongering to raise their profile and stock value."Remember: every time you repeat a story about how awfully, terribly dangerous their products are, you help them raise more investment capital, which is a key input for their business (hooking up statistical engines to money-furnaces)," he wrote in a recent newsletter.However, Steidley said that the world still needs to come up with regulatory framework for AI, and fast."I think we're not quite at the point where we cannot stop AIs from doing things we don't want, because they're not quite more capable than we are, which is to say that we still have time. We still have time to avert these disasters before AI is so smart that it can outsmart us."I think what the world needs is international coordination so no country or AI company in any country builds AI that it can't control that's smarter than humans."That's what we did with the nuclear situation. We didn't just let any country have nukes."Prime Minister Christopher Luxon said he is "concerned" about the dangers of AI, but also sees opportunities.RNZ / Samuel RillstoneWhat's New Zealand's position on all this, then?Prime Minister Christopher Luxon has been a steady advocate for adopting AI within New Zealand.Asked at Monday's press conference if he was worried about AI wiping out humanity, Luxon said "No, no I'm not."Luxon said he felt AI offered enormous potential for New Zealand to boost its productivity, but admitted to some concerns."Yes, we are concerned about the dangers of AI, but I would just say we see the opportunities as being greater than the dangers at this point in time."Luxon said New Zealand "needs to work with like-minded friends to put in place the right regulatory frameworks."He said discussions are ongoing with Five Eyes partners and cybersecurity experts."It's been good to actually hear, even in the last week, that those three big CEOs of the major AI firms are actually saying 'Hey listen, we want to slow this down so that companies and governments can actually keep up to speed.'"Luxon said that businesses also need to be sure they're investing in their own cybersecurity as possible AI-related threats evolve.The government unveiled a national AI strategy last year, which promised a "light-touch and principles-based" approach using existing legislation and regulations, rather than introducing new ones.Dr Andrew Lensen, a senior lecturer in Artificial Intelligence at Victoria University, said then that the strategy was heavy on economic growth opportunities but "worryingly light" on ethical and societal issues."The strategy suggests that new legislation is unnecessary, which I, and many other AI researchers, disagree with," Lensen said.For its part, Labour has promised if elected to government to set up an AI office, an online safety regulator, copyright rules that ensure creators are paid for their work, and impose rules on data centres."Used well, AI has huge benefits," leader Chris Hipkins said."But we cannot leave it to big international tech companies to decide how it shapes our future."