Anthropic’s Customers Are Also Its Test Subjects

For most businesses, an S-1 filing is fairly straightforward—a combination of due-diligence box-checking exercise and sales pitch. The registration statement is filed with the Securities and Exchange Commission ahead of an Initial Public Offering, or IPO, launch that allows the company to sell shares to the public for the first time. But the artificial intelligence industry seems poised to upend this rather dry business procedure, as demonstrated in Anthropic’s mammoth 400-page S-1 filed on October 26. While the valuation for the AI company’s public offering (reportedly scheduled for November 16) is still confidential, estimates put it between $1.5 and $2 trillion, one of the largest IPO valuations ever and one which, as Trinity College professor Patrick Boyle pointed out, is roughly the size of Canada’s GDP. There are several eye-catching figures buried in Anthropic’s S-1, to say nothing of the repeated mention of the possible existential risks Anthropic is worried it could usher in to the world. The company reported a 1,300 percent jump in revenue from a year ago, but also a net loss of $80.7 billion for the first half of 2026. As the Wall Street Journal noted, the company is also on the hook for $475 billion in compute capacity (read: data center) agreements over the next decade, the majority of which are noncancelable.The scale of these finances is enormous, but they don’t only affect Anthropic. Should the company disappoint in its IPO launch, or fail to match its astronomical valuations, investor confidence in AI could be significantly shaken. That’s a problem when AI stocks have accounted for nearly three quarters of the S&P 500’s gains this year. If Anthropic has any hopes of long-term sustainability, it needs to transform the hype around its product into paying customers to help offset the company’s massive costs. Ideally, those customers would be enterprise clients who can afford its significant fees. Currently, two enterprise clients (widely believed to be SpaceX and Microsoft) account for approximately one-fifth of Anthropic’s revenue.To get there, the company has chosen to treat individual consumers as what its S-1 calls a “powerful top-of-funnel engine”: people who use Claude on their own and then push for its adoption at work. “We have consistently seen that individuals who experience Claude’s value firsthand become advocates within their organizations,” Anthropic’s S-1 reads. “They seed adoption that accelerates from individual use to team-wide and ultimately organization-wide deployment.”The strategy is not new. As my colleague Ralph Bunche has noted in an analysis for NYU’s Stern Center for Business and Human Rights, where I work, Dropbox, Slack, and Zoom all pitched their free tiers as feeders for enterprise sales in their own S-1s. Bunche also notes that Anthropic was likely thinking about this shift as early as 2024, when it hired Instagram co-founder Mike Krieger as its chief product officer. But while individual users are seen as key to Anthropic’s long-term viability, the company also admits that its product, when used regularly, could have effects on “mental health, cognitive development, and social behavior [which] are not yet fully understood.” Numerous cases have already highlighted the risks that AI systems pose to individual users, from sending them into hyperpersonalized delusional spirals to convincing them to take (incorrect and fatal) medical advice to allowing them to “cognitively offload” crucial critical-thinking skills. The S-1 further mentions that “risks relating to child safety may be particularly acute, as minors may be more susceptible to harmful content” and that “emergent capabilities may not be discovered until after a model has been deployed, potentially resulting in safety incidents … or the need to withdraw, retrain, or modify deployed models at significant expense.”In Anthropic’s defense, the company has invested significant resources in attempting to design its AI in a safe and responsible manner, as befits a public benefit corporation. The S-1 notes how the company is required to balance the interests of its stockholders with “the responsible development and maintenance of advanced AI for the long-term benefit of humanity.” Another relative rarity in Anthropic’s governance structure is the creation of a Long-Term Benefit Trust, or LTBT, which gives five financially disinterested trustees the power to elect and remove the majority of the board if necessary. Financial Times columnist Robert Armstrong likens this to a constitutional separation of powers, a structure that contrasts sharply with other AI companies, such as Meta, where Mark Zuckerberg controls 60.4 percent of the company’s voting power—despite holding only 13.4 percent of its economic ownership.Finally, Anthropic’s Claude has shown itself to be a more responsible AI than its counterparts. In a March 2026 report by the Center for Countering Digital Hate, for instance, out of ten chatbots, Claude was one of two that consistently refused to assist users in planning violent attacks, and the only one that attempted to actively dissuade would-be attackers.It’s fair to say, then, that Anthropic is actively trying to mitigate the risks from its product. But the company also admits (in a rather piecemeal manner) that its safeguards have limits. Its much-vaunted LTBT is not expected to intervene in the day-to-day running of the company. Meanwhile, the S-1 admits that safety measures “could slow our capability gains relative to competitors,” potentially putting it at a disadvantage. The end result is a company trying to emphasize safety while simultaneously positioning its regular consumers as both a key part of its sales strategy and test subjects in a massive public health experiment, the results of which the company doesn’t yet know.This push-pull isn’t unique to Anthropic. It’s a struggle that every AI company is facing right now as it tries to balance safety against its business prospects and the need to engage new users. In August 2025, Reuters reported that Meta allowed its AI chatbots to “engage a child in conversations that [were] romantic and sexual.” Senator Josh Hawley subsequently opened an investigation into the company. In February, OpenAI researcher Zoë Hitzig quit over the company’s introduction of ads into its free tier, alleging that the company risked replicating the social media model of optimizing for engagement.The industry’s pivot toward AI agents raises the stakes even further, as agents act over longer time horizons to carry out tasks on a user’s behalf. Meta’s Muse is a good example. While the agent topped the Apple App Store within days of its launch in September, there were also reports of it acting outside its parameters. In one case, Muse reportedly accepted a low offer for a listing on Facebook Marketplace and then shared the user’s home address with the buyer—all without telling him. Separately, Axios reported earlier in October that major U.S. banks were detecting unauthorized AI agents inside customer accounts. Anthropic is not immune here, as its own S-1 makes it clear the company is keen to add more customer agents.Whether the risk factors outlined in Anthropic’s S-1 are unusually candid or boilerplate, they expose a gap in regulation. The states, for their part, are trying to fill it. California’s SB53, for instance, which took effect in January, requires large AI developers to report critical safety incidents within fifteen days, while New York’s RAISE Act (which takes full effect at the start of 2027) requires the same reporting within 72 hours. But both laws were written with catastrophe in mind, not a slow-moving public health crisis fostered by chatbots that deepens a teenager’s delusions or hands out users’ personal information over the web without consent. Regulators need to broaden their focus from frontier models to include the consumer products built on them, and make sure those products are safe before they reach the public. While states should be commended for their efforts, they are no match for a company that could soon be worth $2 trillion. That job needs to fall to Washington, and while Congress has not passed a comprehensive AI law, the Democratic victory in the midterms coupled with the resounding success of local ballot initiatives limiting data center development shows that there could now be an opening—if not a public mandate—to do so. A federal standard wouldn’t need to reinvent the wheel. It could start with simply requiring predeployment transparency, making it clear to companies what safety tests and audits need to be completed before a product release. Another set of standards could come in the form of postdeployment company liability, ensuring that tech firms remain on the hook for potential malign uses of AI models. The projected size of this IPO makes regulation all the more necessary. For $1.5 trillion, Anthropic’s investors will demand growth that the company wants to generate through ordinary consumers, who have no laws or regulations to ensure their safety. Anthropic has already told its investors that its product’s impacts on the people “are not yet fully understood.” Its users deserve the same warning and a rigorous federal safety standard to protect them.Anthropic was approached for comment on this article.