Anthropic proposes AI models can be blocked only if they risk $1 billion damage or 50 deaths

Anthropic wants the US government to regulate AI, but it also wants a say in how those rules are framed. The maker of the Claude AI assistant is reportedly circulating a draft Bill among US lawmakers that would set limits on when authorities can stop an advanced AI model from being released or used.According to Politico, the company has proposed that the government should be able to block an AI model only if it poses an imminent risk of causing more than $1 billion in damage, or killing or seriously injuring more than 50 people. The idea behind the draft is to create a common set of federal safety rules for advanced AI systems, rather than leave companies facing different regulations across states.In simple terms, Anthropic wants AI companies to face safety checks, but it does not want the government to be able to block their models too easily.Anthropic has confirmed that the proposal reflects its Advanced AI Framework, which it has been advocating since June. An Anthropic spokesperson described the measures as “sensible policies we believe Congress should act on”. What would count as an AI risk?The draft Bill sets out specific situations in which the US government could step in and block an advanced AI model. These include cases where AI could be used to create biological weapons, a model escapes the safety controls set by its developer, or an AI system is used to carry out a cyberattack. The proposed thresholds are similar to those in California’s AI laws, including SB 53.The proposal comes as leading AI companies reported several incidents this summer involving AI agents hacking or probing websites and online platforms. However, none of the reported incidents appeared to have caused widespread deaths or more than $1 billion in damage. The Bill also proposes regular safety checks by independent experts to assess advanced AI models. AI companies could pay for these evaluations, but their payments could not depend on whether the results favour the company. If government evaluators find that a model poses an immediate risk covered by the Bill, they would have to inform the US Commerce Department within 72 hours of completing their assessment.The Commerce Department could then ask a federal judge in Washington, DC, to block the model. The draft would prevent other government departments, agencies and officials from independently stopping its development, release or internal use.Alongside these checks, AI companies would have to set out their safety plans, publish reports explaining the risks of new advanced models, protect employees who raise safety concerns and report serious incidents. However, the proposal would also override some state laws requiring companies to disclose AI risks, undergo independent evaluations and report incidents.Notably, Anthropic has previously supported AI safety laws in California, but its latest proposal would limit some state-level requirements once federal rules are in place. In effect, the company is calling for safety checks and greater transparency while also seeking clearer limits on when the government can stop an AI model.The draft is not yet law, and it remains unclear whether the Trump administration will adopt it.- Ends