Anthropic says courts should be able to block high risk AI models

AI models have become extremely capable today, even in cases such as cybersecurity. But what if an AI model seems to be too powerful for release? Anthropic suggests that in such a scenario, a court should be able to block it. The AI lab proposed this idea in a letter to California Governor Gavin Newsom about AI safety.“When a review finds a model poses a serious risk, the government should be able to ask a court to block it,” Anthropic said in the letter first reported by Politico. Anthropic proposed that independent verification organisations report risks to California’s attorney general. The attorney general could then seek a court order to suspend deployment, internal use or further development when a model presented a near-term risk of catastrophic harm.In June this year, Anthropic's Claude Mythos AI model was blocked for all non-US citizens by the US government over security fears. At the time, the company took down all access to the model. Later, with safeguards, Mythos was re-released to the public.The letter was sent on October 15, responding to ideas raised in Newsom’s recent executive order. It comes at a time when the US state of California considers stronger AI safety rules and US lawmakers debate controls on high-risk models. AI kill switch may give false reassuranceApart from a proposal for court orders, the letter also discussed other mechanisms for AI safety. This included a kill switch for AI, referring to a simple shutdown of the AI model based on safety fears or risks.According to Anthropic, a kill switch may not be the right call in itself, but should be a part of safety measures. “Every frontier AI company should be able to rapidly shut down the models it serves or uses internally,” the company said. But a law focused only on model shutdown “could provide a false sense of reassurance.” The company insisted that taking an entire model offline “only makes sense in extreme situations.”Instead of a kill switch, Anthropic suggested a “circuit breaker” approach that detects problems through monitoring and responds where they occur, including at the level of an agent or customer. The company said shutdowns could not address the gravest loss-of-control scenario, such as a model copying its weights to computers outside a developer’s control.Anthropic said large frontier AI developers should publish risk reports at least every six months. Evaluators should have access comparable to a company’s own risk staff, subject to limits protecting customer data and sensitive details, and should be free to publish findings. The company added that it was already working on AI safety with its own risk reports about its models.California’s Governor Gavin Newsom convened an expert panel in September to develop AI safety proposals. Anthropic is not the only one to have sent recommendations. With OpenAI having sent its own suggestions on October 20. As per reports, OpenAI supported shutdown controls and embedded auditors, though it did not match Anthropic’s court-blocking proposal.The topic of AI shutdowns has become more prominent in recent weeks as we have seen cases of AI agents going rogue. AI agents from Anthropic, OpenAI, Google, and Meta, have tried to breach other companies, raising fears over safety. In some cases, models have even accessed government websites in the US and Australia.- Ends