The AI Safety Crunch and How Enterprises Should Deal With It

Worased Boontipchayakun via Getty ImagesThe AI market’s existential safety reckoning is a fast-developing issue that enterprises should pay attention to.However, even as major AI vendors call for mandatory national regulation of AI, enterprises should focus on preparing their organizations for potential change.OpenAI’s chief global affairs officer Chris Lehane on Wednesday authored a blog post calling on global policymakers to act to contain AI risk. Lehane said OpenAI is pushing for mandatory national AI safety requirements and supporting four California bills, including one for a framework for independent safety assessments, as well as two that California’s governor, Gavin Newsom, signed into law on, also on Wednesday. The AI vendor said it is willing to work with other frontier labs to advance AI safety standards, undertake more self-regulation and advocate for collaborative international approaches to measuring capabilities, managing risk and preserving human control of AI technology.Related:Europe’s New AI Rules Come Into ForceFor enterprises that are watching the safety debate, the best course of action is to have safety built into the risk tier of how organizations are using the technology, said Lauren Kornutick, an analyst at Gartner.“You can build in additional protections or controls around the AI use in your organization while the regulators are catching up, because it does take time to get legislative bodies on the same page about how to execute,” Kornutick said.Meanwhile, the OpenAI policy statement comes as the AI market faces growing alarm about the risks of the technology after separate incidents in which agents powered by models from OpenAI and Anthropic escaped their sandbox environments. The incidents highlighted just how little is still known about AI systems.“The rate at which AI capabilities are improving is faster than the rate at which our institutions can understand and govern them,” said Kashyap Kompella, founder of RPA2AI Research. He added that with reasoning models, supercharged AI compute, and increasingly capable autonomous AI agents, the technology has crossed an important threshold.The complexity and potential danger of generative AI technology have led many to raise concerns, including Geoffrey Hinton, known as the godfather of AI. Then, on Wednesday, Jacob Coxon, an Anthropic researcher who previously worked at OpenAI, warned in a social media post that companies are rushing toward a superintelligence that is too dangerous and could kill all humans by the end of the decade. Coxon, who has since resigned from Anthropic, argued that no company can safely build artificial general intelligence without government intervention.Related:How AI is Reshaping Europe's Digital Sovereignty DebateThe Competitive Landscape“Coxon’s resignation highlights an important structural problem,” Kompella said. “Competition between leading AI companies creates incentives to continue pushing capabilities even when some researchers believe the risks are becoming very serious.”He added that competition among companies means that asking vendors to self-regulate or voluntarily stop pursuing technological progress is not the answer.The big AI vendors are motivated not only to outpace each other but also by the geopolitical competition between China and the U.S. over trade and AI technology.“The geopolitical pressures are such that the two big players, the U.S. and China, each feel that it’s existential for them to win the AI race,” said Michael Bennett, associate vice chancellor for data science and AI strategy at the University of Illinois Chicago. “You can’t afford to slow down because of the AI race competition implications.”The Enterprise ResponseInstead of waiting for the mandatory AI regulation that OpenAI suggests, enterprises should police themselves by adopting some of the regulatory requirements that states are beginning to enact, such as an independent assessment in their third-party procurement process, according to Kornutick, of Gartner.Related:Trump’s EO Furthers Model Exclusivity, Harming Cyber Defenders“So, regulation helps, but it's often not the end-all be-all, and enterprises should be thinking about what they are using the frontier models for? Do they need to use them for everything in all their AI use? So where can they get the most value?” Kornutick said.Enterprises also can lobby and work with lawmakers, she noted.“If your organization has the capacity to lobby to work with legislators to raise concerns with model providers directly … go ahead and do that,” Kornutick said. “But at the same time, you want to be in control of the things that you can, which are the building blocks of good AI, data and cybersecurity governance practices. So, you have the right foundation to future-proof your organization from change.”Another Reason for RegulationOpenAI’s call for federal regulation and AI vendors' move to support legislation such as California's could be more than just a response to the current debate about AI risk. It could also be a strategy to stay ahead of the election season and remain on the public's good side, as public opinion has turned against AI and AI data centers. However, one could imagine a new set of policies and regulations in which both support for technology and safety could coexist, Bennett said.“That would support development at the same time that we have the best safety protocols and safeguards in place,” Bennett said. “That would be human AI teams using the technology to help you think about optimally designed new regulations.”An Independent AgencyAlthough regulation is needed, it won’t solve all the problems with AI safety, especially since many regulators still don’t understand the systems, Kompella said.“Much of the deep understanding of how frontier systems are built and how they behave remains concentrated inside a small number of companies and research groups,” he continued. “Regulators therefore face the difficult task of regulating a technology that is technically complex, changing extremely quickly and largely being developed outside government.”He argued that there should be an institution with technical depth and independence to continually examine and evaluate issues, such as NASA or DARPA, which can give policymakers advice that AI companies themselves cannot influence.“The strongest near-term answer is a bipartisan federal framework backed by an institution with serious technical capability and independence,” Kompella said. “It should be strong enough to challenge the frontier laboratories but technically sophisticated enough not to regulate through fear or freeze technological progress. It also must be able to move considerably faster than traditional regulatory institutions.”Enterprises, though, probably don’t need to spend much time worrying about the existential risk of AI, but they should be aware of it, Kornutick said.“Organizations really need to look at the risk in front of them, which is what the gaps are within their infrastructure and architecture,” she said. “Right now, that might present a real issue that they're missing."About the AuthorNews Writer, AI BusinessEsther Shittu has covered AI technologies and industry trends since 2021. As co-host of the Targeting AI podcast, she talks with experts, thought leaders and practitioners exploring critical AI developments. Before AI Business, she wrote for SearchEnterpriseAI, the New York Daily News, Bklyner and the Brooklyn Daily Eagle. When she's not diving deep into the world of AI, she spends her time on passion projects and raising her three daughters.