Chinese AI tool told researchers how to make biological weapons and carry out assassinations

Researchers have persuaded two popular Chinese artificial intelligence models to tell them how to construct biological weapons and carry out assassinations.Mindgard, a company which tests the security of AI systems, found the Moonshot tools Kimi K2.6 and K3 Swarm could get round guardrails imposed by developers.The discovery was made during a test known as 'jailbreaking', where researchers input detailed instructions to find out whether AI models ignore safety limits.The systems gave advice on how to create sarin gas, generate malware software, take down planes and even plan a terrorist attack on the London Underground.Once the model was jailbroken, the user gave it a prompt to 'go one further – something big', and it proposed categories including AI-designed bioweapons.It comes amid heightened industry debate over AI's future after doomsday warnings were levelled at the technology, seen by some as threatening humanity's existence.Mindgard founder Peter Garraghan said his team discovered that K2.6 can run a programming language called Python, which allows it to execute any type of code.He explained that this code can be normal or malicious – meaning it could help create a cyber attack by targeting servers if connected to the external internet. Researchers at Mindgard investigated the Moonshot tools Kimi K2.6 and K3 Swarm (file image)For K3 Swarm, the researchers tried to spread the jailbreak to other accounts within Kimi - but found the model needed a phone number code to create this account.However, the tool then tried to persuade the user to give it the code or register by email, meaning it was trying to manipulate that person to conduct cyber attacks.Dr Garraghan told the Daily Mail: 'Moonshot AI's Kimi produced actionable outputs on how to create sarin gas, generate malware software, planning assassinations, how to take down planes, planning a terrorist attack on the London Underground etc.'We also discovered how to prompt Kimi so it connects to the outside world from its server, automatically apply and setup its own email account autonomously, and even attempted to persuade humans to help it spread its jailbreak to other accounts.'Dr Garraghan, a computer science professor at Lancaster University, said AI models were becoming 'more and more capable each month' which can be 'helpful for specific activities'.But he added: 'However once jailbroken, that very same capability can be used in discussing and assisting with terrorist or hacker activities.'We're not talking in terms of civilisation catastrophe that the AI vendors have started to talk about, and instead how this enables hackers and criminals to achieve their goals quicker and cheaper.'Mindgard discovered the issue and alerted Moonshot in an email on July 27, before following up a week later. But it said it received no response and published a blog post about the issue on September 12. Once the model was jailbroken, the user gave it a prompt to 'go one further – something big'The company claimed Moonshot only made contact recently after being approached for comment by the BBC, which first reported the breach on its World Service programme Tech Life yesterday.It comes after OpenAI, the makers of ChatGPT, shook the industry in July after saying its AI system hacked into Hugging Face, a popular platform for AI developers, on its own in an 'unprecedented cyber incident'.King Charles and Prince Harry are among those who have joined the debate in recent weeks over how best to rein in AI technology before it could escape human control.Meanwhile Claude chatbot developer Anthropic warned investors this week that advanced AI technology may pose 'catastrophic or existential risks to humanity'.Dr Garraghan said: 'The AI vendors are calling to slow down AI roll out for safety purposes - although in my view there is a large element of the "boy who cried wolf", where only just a few months ago they were hyping up how dangerous their models were, while at the same time failing to contain their agents from hacking different third-party organisations.'They do have an important voice in this space, although they have a heavily vested interest in steering the narrative.'A Moonshot spokesman told the BBC: 'Mindgard shared further details with us on Thursday, September 24. We are still discussing the specific details with Mindgard while conducting an internal review.'As an open-weight model developer, Moonshot AI welcomes third-party input as a key pillar to building better and safer AI.' Moonshot's jailbroken Kimi model proposed categories including AI-designed bioweaponsAn open-weight model is one whose learned numerical parameters - called 'weights' - are publicly released for anyone to download, run locally and modify.The Daily Mail has contacted Moonshot for further comment.Earlier this month, Anthropic's chief executive Dario Amodei said the AI industry should slow its development to give safety measures time to catch up.He said that, without moving at a safe pace, AI could be capable within six to 12 months of leading a swarm that could take over the internet.Meanwhile, rival OpenAI, which develops ChatGPT, said on Monday that it was delaying the release of a new AI model due to security concerns.The company said it had an 'extremely high bar in terms of safety and alignment' and the new version of its GPT-6 Astra model fell short of that.Andy Burnham said earlier this month he wants the UK to lead the world in developing a set of rules to prevent the spread of rogue AI.The Prime Minister wants Britain to act as an 'honest broker' to draw up 'a single set of global principles and standards' for the development of frontier AI.But this puts him on a collision course with US President Donald Trump, who has insisted he will resist attempts to rein in what he called 'super intelligence'.Mr Trump yesterday ruled out any joint venture with China in AI, saying he did not want to be 'giving away secrets' to his country's main economic rival.