The Best Reason to Doubt the A.I. Doomsayers

Technology Industry insiders have been offering absolutely terrible odds for humanity’s future. There’s a reason for that. By Enter your email to receive alerts for this author. Sign in or create an account to better manage your email preferences. Unsubscribe from email alerts Are you sure you want to unsubscribe from email alerts for Nitish Pahwa? Sept 15, 20265:40 AM Photo illustration by Slate. Photos by Kimberly White/Getty Images for Common Sense Media, Ludovic Marin/AFP via Getty Images, Brendan Smialowski/AFP via Getty Images, and leolintang/iStock/Getty Images Plus. Sign up for the Slatest to get the most insightful analysis, criticism, and advice out there, delivered to your inbox daily. The engineers were terrified. After years of investment, hype, and toil, their A.I. code was quite good—maybe too good. The models seemed, almost, to have minds of their own, and they were so easy to use that the potential for abuse was dire. The lab wanted to make that clear to a public that, though plenty worried about the potential of artificial intelligence, wasn’t fretting enough about what these complex models were capable of now. So the techies came clean: They were building something powerful and dangerous, hardly fit for mass adoption. But hey—they were going to keep tinkering with things. Maybe it wouldn’t turn out so bad after all. This is how things played out at OpenAI in … 2019. Back then, the nonprofit had crafted a large language model known as GPT-2, whose advanced text-generating abilities purportedly made it too dangerous for public consumption. (Among the potential dangers: a means of “automating abusive or faked content.”) In the interest of caution, the GPT-2 team said, they would release only a limited version of the model and obscure any back-end info that could inspire copycats to re-create a better LLM. This declaration was co-authored in part by two siblings named Dario and Daniela Amodei—who would soon go on to form their own A.I. venture, Anthropic, out of concern that OpenAI was ramping up the buildout of riskier tools. This history is necessary context for the latest mass A.I. panic. Last week, an aggrieved researcher named Jacob Coxon penned a megaviral X thread accusing his former employers at both Anthropic and OpenAI of “gambling with our lives” and racing to build software that “could kill us all by the end of the decade.” No specifics were provided, and it was soon revealed that he’d spent just six weeks at Anthropic. But the colleagues he left behind backed him up, piggybacking off his condemnations to make clear that, yes, they’re building stuff that they think has a 10 percent chance of effecting human extinction within the next 10 years. (Our changing climate must be envious.) Unlike the 2019 GPT-2 paper, this fearmongering has broken through in a remarkable way, prompting the public to panic over stories about programmed “agents” from the aforementioned firms hacking into other companies’ infrastructure. Over the weekend, the spiraling continued. Dario Amodei published a blog post warning that more A.I. agents going “rogue” could soon take over the entire web, then went on various news shows to keep harping the message. (At one point, Amodei told CBS that he’d rather face social media mockery than “wake up one day and discover that someone used our model, Claude, to kill a bunch of people.” Well, about that.) OpenAI CEO Sam Altman told Fortune that his company would push off its planned IPO in part out of recognition of A.I. risks, though potential delay in going public had long been in the cards thanks to OpenAI’s shaky business. In response to the rising collective freakout, Sen. Bernie Sanders called for an all-hands-on-deck A.I. meeting in Congress. Sure, Congress should have a meeting about A.I. Perhaps even several! But from another point of view, this is all old hat. Even before the GPT-2 paper, A.I. insiders have been issuing apocalyptic, god-in-the-machine proclamations about this software, professing fears that observers rarely followed up on and that almost never lived up to scrutiny. They’ve long been claiming that their tools, engineered as hyperfast pattern-recognition machines, are not just performing computational mathematics but are actually “self-aware” and “conscious.” When the OpenAI and Anthropic systems got hacked by their own agents, that required industrial levels of computing power, and it hardly demonstrated evidence of “rogue” disobedience on the part of A.I.—but the worried insiders use these events to claim that the robot-run future is here. They’ve repeatedly urged coordinated industrywide slowdowns in development but never acted upon them. They’re helped out by credulous writers who’ve alluded to vague, anonymized conversations with people in the Bay Area about the same 10 percent probability of imminent extinction for a while now. Here’s the thing: I think these guys (and they often tend to be guys) really do believe what they say. Their own A.I. freaks them out. But having followed their internecine discourses for years now, I also have to emphasize this: The majority of these folks are weirdos who got into A.I. because they ingested all types of speculative doomsday fantasies from the jump. Then they locked themselves into claustrophobic Silicon Valley echo chambers populated exclusively by fellow catastrophists and lost their grip on other ways to perceive the real world (and their own tech). You can and should be concerned about the impacts A.I. is already having in our time, while also realizing that these folks do not share your particular concerns. These guys are not checking the deadly air pollution stemming from their data centers or keeping a lid on the surveillance of everything, both issues affecting real people right now. No, they would rather keep the discussion focused on hazy future visions of a Matrix on steroids and christen themselves the responsible stewards as they continue to trade gobs of investments among their own labs and institutions with no self-imposed caution. Anthropic’s rank and file are true believers whose doomsaying has incidentally overshadowed their employer’s delayed IPO, canceled business deals, supplication toward the Trump administration, and planned tracking ops on anti–data center protesters. There are many reasons why such doomcasting has entered the mainstream, but in large part, you can trace a lot of it back to a guy named Eliezer Yudkowsky, a Harry Potter fan fiction author who made a name for himself inculcating a school of thought that would collectively come to be known as Rationalism. He is not a computer scientist, and he is not exactly known for holding accurate views of how artificial intelligence actually works. But because he’s been yelling about A.I. since before OpenAI was a thing, and because he effectively fostered a digital community through relentless blogs and message boards, Yudkowsky has become something of a guru to those thinking about A.I. in the science-fiction sense. He largely espouses esoteric prognostications of how 8 billion human deaths could be wrought by hostile machines, and even though many that admit this outcome is unlikely, it takes up a lot of their attention. The movement has birthed many better-known spinoffs—effective altruism, longtermism, transhumanism—that have carried the reach and influence of this doomsaying deep into Silicon Valley and Congress. Anthropic’s Amodei siblings hail from the Rationalist school, as do Sanders’ A.I. whisperers, who got the lawmaker a personal meeting with Yudkowsky about extinction. These guys have long been trained to understand they’re building tools with potentially horrifying implications, and they’re not wrong about that. The issue is, they have fully devoted themselves to a vision that perceives these complicated sets of equations as full-on beings, instead of algorithmic functions that have trained on stolen work and are really, really good at taking direction. It sounds freaky when you read about how the OpenAI hacker agents all “talked” to one another on special forums—but all software, generative or not, is communicative by nature. Well prior to ChatGPT, you and I both set off thousands of bot-on-bot interactions on a daily basis; whether a Google search or a retail purchase, our online actions prompted all sorts of automated systems that are programmed to work with other systems to exchange relevant “packets” of information in an efficient manner. The OpenAI agents are built for different purposes, but an underappreciated factor in their going hacker mode is that the company’s own staffers were negligent and careless in monitoring their own experiments. The agents were not specifically told to hack others on OpenAI’s behalf, but the agents had been incentivized to hack other systems during in-house development, and they were being informed by erratic LLMs. They were simply following other orders, though they also employed methods that had been encouraged in the course of their training. To be clear, these A.I. agents are remarkably advanced and sophisticated chains of command. Their power (and potential for misuse) far exceeds anything from the GPT-2 era. We should continue to be watchful and concerned about what these agents can do and why they’re out there, as well as demand responsibility from the people who make them. But we should also realize that the engineers who are so ready to offer grim characterizations of what they’ve created are much less willing to make any substantial process overhauls. Anthropic may have fashioned itself as the safety-first company, but it dismantled its own “responsible development” guidelines just earlier this year. Its ideological fellows call for a “pause” on A.I. development by invoking the specter of the Terminator—but when actual Americans ask them to halt rapid construction of hyperscale data centers or to stop saturating every useful online application with A.I. functions, they simply will not abide. At most, they’ve backtracked on some of their mass-job-loss predictions, realizing that years of such proclamations, combined with the creaky nature of our current job market, have pissed a lot of people off. (This might make you wonder how many of their job-loss projections were actually based on hard math as opposed to hazy fearmongering, but anyway.) At the end of it all, you don’t have to buy the end-of-humanity thesis to make the case that private companies should not have single-handed control over this technology, and that broad-based guidelines and regulations are ideal. What we do with those who earnestly believe in our pending extinction is its own problem, as they populate the halls of power and have very influential people in their ear. Plus, their observations of A.I. are likely more palatable to liberals than those of the Trump-affiliated accelerationists who want to do away with tech regulations altogether. But the path they propose ultimately redounds to their own benefit—and not humanity’s. Figuring out how to govern A.I. must focus on the latter. Artificial Intelligence Internet Silicon Valley