Woman used Claude as diary, Anthropic alerted police and got her arrested

A US woman used Anthropic’s Claude chatbot like a personal diary, but the conversation eventually led to her arrest. The woman reportedly used the AI chatbot to document her everyday thoughts. However, in one such entry, she allegedly made threats against a Florida sheriff’s office. The messages were flagged by Anthropic’s safety systems, and later led to her arrest. The woman, identified as 30-year-old Carli Michelle Heller from Bonita Springs, Florida, was reportedly using Claude to write down her thoughts. However, in one such entry on September 26, she allegedly wrote about planning an attack on the Lee County Sheriff’s Office. “I’m going to shoot up the sheriff’s right the now,” the message read, according to the arrest report, as reported by WINK News.The following day, at around 1.07am, another message was reportedly sent from the same Claude account. In it, Heller allegedly mentioned having a new gun and wrote: “This is 100% last chance I’m done. I got a new gun today. you.” Given the nature of the alleged threats, Heller’s conversation with Claude was flagged by Anthropic’s safety systems and escalated to a human review team. According to the arrest report, the team then contacted law enforcement.Following the alert, deputies went to Heller’s home in Bonita Springs and detained her without incident. An intelligence detective with the Lee County Sheriff’s Office later took over the investigation. Heller has been charged with making a written threat of violence. How did Anthropic alert the police?AI chatbots such as Claude and ChatGPT are designed to be helpful and conversational and are private to an extent. However, these platforms also have safety systems in place to detect potentially harmful or prohibited content. In some cases, these systems can flag a conversation for further review, with human teams stepping in when the content raises more serious concerns.Anthropic’s policies prohibit using its products to facilitate or promote violence or intimidation. The company also has a process for responding to government and law-enforcement requests. In limited emergency situations involving a risk of death or serious physical injury, Anthropic says it may disclose user information to law enforcement.This does not mean every troubling conversation with an AI chatbot is automatically reported to the police. However, when a conversation involves what the company considers a credible and serious threat, its safety and emergency procedures can come into play.In this case, the threatening messages were first picked up by Anthropic’s automated safety measures. Given the nature of the alleged threats, the conversation was then escalated to a human review team, which contacted law enforcement, according to the arrest report.- Ends