Did AI really ‘go rogue’? Not exactly, but we have plenty to fear

OpinionChief scientist, AI Institute, UNSWJuly 30, 2026 — 3:30pmIt’s an ancient fear. It goes back thousands of years. You find it in the story of Prometheus in ancient Greek mythology. And in stories like that of Surtr in Norse mythology. What if our inventions and discoveries get the better of us?Wind forward to today, and people are starting to fear artificial intelligence going rogue and getting the better of us. OpenAI, for example, just published a blog post describing how its latest and most capable AI agents went rogue and hacked into Hugging Face, one of the most popular websites for AI researchers to share resources.Who’s in command, and can laws make AI firms accountable?iStockIn response, a bill has just been proposed in the US Congress to force AI developers to provide a “kill switch” to shut down any AI that goes AWOL.Now, there’s plenty to worry about, but not what you or Congress might think. And it’s certainly not what much of the reporting would have you think. News articles have talked about an AI that went “rogue”, that “escaped” its confinement, and got “outside”. No AI in this case went rogue.Indeed, the whole problem was that the AI did exactly and only what it was asked to do. It was asked to solve a cyber challenge, which it did rather too diligently.And no AI “escaped” and got “outside”. The reality was much more prosaic. The AI was put in a sandbox and given limited web access. So it worked out a “hack” to get unlimited web access. The reality is that OpenAI put it in a rather insecure sandbox.And as far as we know, the AI agent never left OpenAI’s machine. It didn’t escape and get outside. It merely accessed information that it shouldn’t have.Of course, OpenAI has every incentive to talk up the capabilities of its latest AI. It surely won’t hurt its forthcoming initial public offering on the share market. Or in attracting new customers to its frontier models and away from competitors such as Anthropic. And it might also help negotiate some favourable regulation in the ongoing AI race between the US and China.Personally, I’m more worried about the AI bubble bursting. We’re still seeing trillions of dollars of capital expenditure on data centres, yet AI is delivering value much slower than this rate of investment supports.Circular financing deals – such as Nvidia investing in OpenAI, which spends this investment on Nvidia’s chips – don’t ease my worries. The Australian Bureau of Statistics’ latest figures suggest this technology capital expenditure is keeping the national economy afloat, and keeping us from recession. But it can’t continue.The lesson I take from the Hugging Face hack is that the cyber capabilities of the latest AI models are very impressive and very troubling. Rather than reason its way to a solution to the cyber challenge, OpenAI’s agent worked out the answer was hidden on Hugging Face’s website, protected by a password. So it gained access to the internet, found a stolen password and accessed the answer.OpenAI’s agent could have been turned off easily at any time. Just reach for the power switch on the side of the computer. We don’t need to regulate for kill switches. We already have them on every computer on the planet. But what we don’t have is mechanisms to prevent these AI models falling into the hands of bad actors and being used to do harm.And this problem only got harder with China and others producing “open-weight models” like Kimi 3, which are free to all. This genie is out of the bottle.Governments, therefore, should be introducing new regulations. We need, for example, to hold tech companies more liable for their products. If a car manufacturer produces a car that catches fire, we hold it liable. Ford discovered this to its cost in the 1970s with the scandal around the Ford Pinto’s fuel tank.Why, then, aren’t we holding AI companies more accountable for the harms they are causing? The Digital Duty of Care under discussion in the Australian parliament is one place to start. This legislation needs to be expedited.We also need to hold companies more accountable for facilitating the distribution of harmful content. How is it that Google Search can help you download nudify software? Or ChatGPT offers advice about self-harm that helps people self-harm? These wrongs were entirely predictable and preventable.Finally, we can’t depend on the goodwill of companies to report these incidents and regulate themselves. Social media platforms have demonstrated repeatedly that they can’t be trusted to mark their own homework.When Anthropic decided not to release its latest Fable model because of its impressive cyber capabilities, it instead released it to a select group of companies to start identifying and patching their software. All very good and responsible.This group included several banks. Yes, our bank accounts are in danger of being drained by hackers supercharged by AI. But Anthropic only released the AI model to banks in the United States. No banks in Europe or Australia or Asia were given advance access. How are they supposed to get ahead of the hackers if they’re left in the dark?We cannot then continue to rely on tech companies doing the right thing. It’s time governments stepped up. I was pleased to hear Prime Minister Anthony Albanese give a major speech on artificial intelligence, saying our government would be more interventionist, and would ensure AI was used for the public good.It’s time to turn those promises into action.Toby Walsh is chief scientist of UNSW Sydney’s AI Institute. His is the author of God AI: Boom or Doom? What to expect when the machines outsmart us, to be released on September 1.If you or anyone you know needs help, call Lifeline on 13 11 14 (see lifeline.org.au), Beyond Blue on 1300 22 4636 (see beyondblue.org.au) or 1800 RESPECT (1800 737 732).Get a weekly wrap of views that will challenge, champion and inform your own. Sign up for our Opinion newsletter.From our partners