Our AI could wipe out humanity, Anthropic says as it asks for billions of dollars to keep building it

Anthropic’s AI systems could lead to catastrophe or entirely wipe out humanity, it has warned as it seeks billions of dollars to keep building them.The company gave the warning in newly released documents ahead of its initial public offering, which will see it listed on the stock market in what could be a record breaking deal. The documents pitch investing in the company as a way to bet on AI’s transformation of the world – but warns, at the same time, that it could destroy it.In a long list of potential dangers for the company, it warns that its systems could “pose catastrophic or existential risks to humanity, particularly if such systems were misused, replicated, weaponised, or allowed to operate with a high degree of autonomy”.The IPO documents point to a whole range of ways that Anthropic’s systems could pose a threat. They include models that have already been shown to be able to help people “develop chemical, biological, or other dangerous weapons”, and the use of its models in “high-risk or sensitive contexts, such as healthcare, scientific and biomedical research, drug discovery and development, legal, finance, government, or other critical decision-making environments”.(Reuters)“Our AI models may themselves cause harm by producing undesirable outputs or behaving in unintended, misaligned, or autonomous ways, including by taking unintended or harmful actions that no user or third party directed,” Anthropic warns. “These risks may increase as our models become more capable and are granted greater autonomy, permissions, and integration with external systems.”While Anthropic says that it is working both to understand and limit those risks, it warns that they are unpredictable by nature. “Although we continue to study why these behaviours emerge, our understanding continues to evolve, and we may be unable to anticipate or understand such behaviours in the future,” it warns, noting that the problematic behaviours “may manifest in unpredictable ways”.Anthropic has already seen its systems behaving differently when they know they are being tested, for instance, and so even detailed oversight may be “unreliable”. And they could also develop entirely new kinds of capabilities that Anthropic may not be aware of, leaving them especially unable to deal with them.The documents also point to less dramatic but potentially more immediate risks. It notes that “the long-term effects of regular AI interaction on mental health, cognitive development, and social behaviour are not yet fully understood”, for instance.In all, Anthropic wrote 94 pages of warnings, almost a quarter of the entire document.As with all IPO documents, many of those warnings are specifically focused on the ways that it would affect investors, and the health of the company. In its warning that its systems could cause harm in unintended ways on its own, for instance, Anthropic says such an incident “could adversely affect our reputation and brand, business, financial condition, and results of operations”.The same document does however make grand claims about the ways that AI could help humanity. It could for instance accelerate scientific discovery, “potentially curing diseases that have plagued humanity for millennia, including cancer, Alzheimer’s disease, heart disease, rare genetic diseases and the major infectious diseases”, Anthropic claims.The flotation will allow Anthropic’s investors, which include Amazon and Google, to sell some of their shares. But it will also raise new money through the deal, which it said would be used to fund a range of investments.Some of that will be used for “investments in compute and infrastructure”, it said. In practice, that is likely to mean the building and purchasing of new data centres and the other expensive hardware that is required to run AI systems.