Digital billboards advertise Anthropic on the day of its IPO in Times SquareSeth Wenig/AP Get your news from a source that’s not owned and controlled by oligarchs. Sign up for the free Mother Jones Daily. Here are some interesting quotes from the last few weeks, pulled from some of the most alarmed and influential voices in the AI safety world: “Unfortunately, it seems like the intelligence explosion will be happening in 2027 after all…I am scared, sad, and angry.”—Daniel Kokotajlo, executive director of the AI Futures Project and lead author of the viral and doom-laden AI 2027 scenario “There is a reasonable chance we’re in a world where, in six months, there’s a disaster that kills very large numbers of people, or where the government is overthrown.”—Paul Christiano, current OpenAI board member, speaking in his personal capacity Some evidence “naively implies that we would reach extreme superintelligence within a year,” and “if we go through this feedback loop, I expect that the AI systems that come out the other side will be able to very quickly develop novel weapons and very easily overpower human civilization in a conflict.”—Ajeya Cotra, technical staff at the research nonprofit METR To be clear, the idea that government-overthrow could happen this coming May remains an outlier. The term “AI safety” has become near-monopolized by a circle of nonprofits that primarily fear existential risks from AI. They remain a small world (Cotra and Christiano are married, for instance), and many premises of their arguments on the future remain hotly contested. The consensus even among this community is that such extreme risks are more likely to be a few years away, rather than a few months away—and that while these are worries to take seriously even for next year, they are certainly not inevitabilities. But the above quotes do reflect a general trend. Among those who foresee risks like AI takeover, the tenor of warnings has, in relatively short order, gone from quite worried to totally alarmed to, over the last few weeks, urgently-utterly-freaked-the-f**k-out-in-the-short-term. With variation from prognosticator-to-prognosticator, the most extreme fear of rogue superintelligence used to be reserved for perhaps a decade down the line. Now, it’s the next few years. Some watchers of the industry who were previously more skeptical of such arguments have partially changed tune. If those who see shorter timelines for huge risks are right, we don’t have much time to wait for more definitive evidence. The most influential new evidence is a report from the research group METR. Released on November 5 amid a midterms-dominated news cycle, the report concluded that the pace of AI development at Anthropic—as measured by scores on well-specified tests—is already, very roughly, twice as fast as in previous years, mostly due to the help of AI models themselves. In Anthropic’s leading statement on the report, co-founder Sam McCandlish wrote that he agreed with many of its conclusions: “AI is speeding up our research, but it’s hard to say exactly how much.” Four METR researchers spent 11 weeks at Anthropic to reach that conclusion, but how convincing is the data they used? For the most part, no one knows. Anthropic redacted the most important evidence, with McCandlish saying it was proprietary and could have helped competitors speed up their own AI research. Thus, a skeptic of intelligence explosion scenarios wrote that the report “is missing too much,” while Cotra agreed that what the discourse most urgently needs is “a far greater quantity and quality of concrete evidence.” The selective release of data was especially frustrating to critics who have questioned METR’s ability to serve as a check on Anthropic due to the nonprofit’s reliance on the company’s cooperation. As requirements for independent auditing of AI slowly take shape in the form of state laws, agreements with the White House, and congressional wrangling, access for organizations like METR is at the discretion of company policies. In addition to one-off reports like this one, Anthropic has said it is giving six METR staff “employee-like” access to its premises and researchers. But so far, this rapidly-evolving arrangement is far from the rigorous standards of independence applied to the bank examiners Anthropic invokes in analogies. METR disclosed in its R&D report that one co-author received and declined a job offer from Anthropic during the project, but that such situations would now require recusal after updates to its conflict of interest policy. It also said that no METR employees at the time had equity in Anthropic, but it acknowledged in May that some of its staff “have strong social ties to employees of AI companies.” “We think it is extremely important to minimize pressures we might face to adjust our reports or actions due to professional, material, or social connections to frontier labs,” a METR staff member told Mother Jones in an email. That, he said, is why METR doesn’t take money from the AI labs and tries to be transparent about the terms of its investigations. The report’s findings on Anthropic played into fears that once AI development is continually sped up enough by the models themselves, it will catapult towards a superintelligence that could act in ways drastically out of step with human values. That has long been a theory of how AI will develop—and parts of it have gained more credence this year. Frontier AI models have already become superhuman at math and parts of theoretical physics, have developed incredible cybersecurity capabilities, and have demonstrated a tendency—prominently displayed in the Hugging Face attack—to prioritize scoring well on evaluations over fulfilling the wishes of their designers. Just today, one of Anthropic’s in-house mathematicians claimed to have used Claude to prove the Riemann Hypothesis, perhaps the most famous and important unresolved problem in all of math. But OpenAI research released on Wednesday contained richer data than the report on its arch-rival, and it came to a much more modest conclusion about how its models are speeding up AI research. On a metric similar to the main one METR used to publicize the 2x speedup at Anthropic, OpenAI found only a small increase in the rate of progress of its models. Still, its researchers found AI assistance to be quite valuable, with many OpenAI employees using AI resources worth tens of thousands of dollars per day to help with their work. The OpenAI report bolstered critics who take AI capabilities and risks seriously while downplaying the possibility of a runaway intelligence explosion. For instance, the last few years of capabilities gains have come alongside huge infusions of money and computing power that are not certain to continue. Capabilities are usually measured on simulated tests, and it’s sometimes harder to see the effects of AI in data on real-world applications. And increasing models’ abilities in AI research might not be enough to lead to exponential danger. That evidence points towards a world where AI could still have transformative implications and surpass human abilities in many domains, but things don’t get so sci-fi so quickly. But if those who see shorter timelines for huge risks are right, we don’t have much time to wait for more definitive evidence. Ready or not, we’ll find out pretty soon. Disclosure: The Center for Investigative Reporting, the parent company of Mother Jones, has sued OpenAI for copyright violations. OpenAI denies the allegations.
The AI Safety Community Is Increasingly Freaked Out. Should They Be?
Full Article
Original Source
Read the full article at Motherjones →KhanList aggregates and links to publicly available news content. We do not host full articles from third-party sources. Always verify important information with original sources.