The Flaw at the Heart of Anthropic’s I.P.O.

What I’ve got here is Anthropic’s 300-plus-page prospectus that the company filed before going public. A lot of this document, nearly 100 pages, is outlining very real risks that Anthropic sees — not just business risks to Anthropic but risks that its technology has the potential to cause for all of humanity. And those risks are existential, and I mean really existential — like, potentially wiping out humanity. “I won’t lie to you. There are real dangers.” But Anthropic thinks that it has a solution for this. Instead of organizing itself like a traditional corporation, where it’s acting in the interests of its shareholders —— “Directors represent the stockholders and are responsible for the way the business is run” —— Anthropic has organized itself as a public benefit corporation, acting in the interests of humanity writ large. That’s what the co-founders wanted when they left OpenAI to start Anthropic. But what worries me is that the company is asking us to place a lot of faith in them to be able to manage these very real, very scary risks, and I’m just not sure that it’s capable of doing it by itself. So let’s look at the company structure first. It’s pretty novel. The real power within Anthropic lies with two distinct groups. Its seven co-founders have a majority stake in the company, collectively, of 50.1 percent. That includes Dario Amodei, the C.E.O. of Anthropic; his sister, Daniela, who’s the company’s president; as well as five other members of the founding team, who all have roughly equal shares in the company and have control of it and also the capacity to direct its day-to-day operations. The second locus of control within Anthropic is this long-term- benefit trust. This includes five trustees — four currently; one of the spots is open. And you should think about it almost as a group of wise elders that touch a lot of industries that Anthropic is related to, things like public health or physics. What sticks out to me as an economist is that this group includes Ben Bernanke, the former chairman of the Federal Reserve Board, who guided this country through a financial crisis and has a deep array of research expertise in moments of crisis. “I spent a great deal of my academic career studying not just the Great Depression but the whole general issue of how financial crises and financial distress affect the overall economy.” Public benefit corporations are rare but not unheard of. Companies like Warby Parker are structured similarly to act, at least in part, in the interests of the general public. What’s unique about what Anthropic has done is they’ve given some teeth to this idea of acting in the public interest through the long-term-benefit trust. This group doesn’t day-to-day manage Anthropic or its operations. Instead, what it does is elect a majority of the company’s board of directors, with the goal of making sure that Anthropic is keeping humanity’s greater good in mind. Right now, the trust has appointed three of those directors. One seat is open. The other three seats are held by Dario Amodei and his sister, Daniela, as well as the venture capitalist Yasmin Razavi. They are elected by shareholders. But even this structure — while unique, novel and important — just isn’t enough to meet the moment. The long-term-benefit trust moves inherently slowly, and in the world of A.I., what we’ve learned already is that things move fast and the capacity for misaligned agents or cybersecurity threats is very real. I spent many nights with this prospectus, and let me tell you, it is thick. This document from Anthropic contains information like the state of U.S.-China tensions and the race for artificial intelligence. Also, how much the company is spending annually on security for its C.E.O., Dario Amodei. That’s $3 million a year. It’s quite a lot. And also who Anthropic views as its real competition: Google and OpenAI. Sorry to Elon. He did not make the cut, but Anthropic devotes nearly 100 pages of this document to risk factors that it thinks its investors should know about before they invest in the company. They break these risk factors up into basically three categories. First, business risks. These are the kinds of things that are generally highlighted by companies in documents like this one. A.I. is an intensely competitive industry, and Anthropic is telling us explicitly what it needs to stay at the frontier of artificial intelligence development and how that’s not a foregone conclusion. This prospectus also outlines policy or regulatory risks. For example, it talks through the potential consequences of some of the data center moratoriums that will impact its capacity to develop, compute and then effectively train its models. It also talks about the company’s altercation with the government over its stance that its technology shouldn’t be used in the development of mass surveillance technologies. It’s kind of weird, actually, that this prospectus, which is meant to be about educating investors about potential risks that the company faces, spends so much time talking about the risks that the technology presents to humanity. In fact, it’s explicit: “Our A.I. models might behave in unintended, misaligned or autonomous ways, including by taking harmful actions that no user or third party directed.” That’s wild. The nightmare scenario for six to 12 months would be that you have a swarm of these agents that form some kind of botnet collective and could take down large segments of the internet. But when you talk about what that could mean to the financial system, what that could mean to our water systems or our critical infrastructure, things start to get really scary. I agree with the Anthropic prospectus that those types of risks pose real concerns for the company, for its reputational harms and potentially for its investors who stand to lose money. That’s why Anthropic’s leaders have been calling — really begging — for more regulation and more involvement by the government to pace development of this technology, in which misaligned agents could easily reap significant economic harm not just for investors but for all of us.