Why the Next Bioterrorists May Not Make the Same Mistakes as Aum Shinrikyo

Why the Next Bioterrorists May Not Make the Same Mistakes as Aum Shinrikyo

In conversations with law enforcement and security professionals, one view has repeatedly emerged: Aum Shinrikyo’s failed biological weapons program in the early 1990s shows that the risk of bioterrorism is overstated, despite warnings about AI-assisted bioterrorism and calls for enhanced preparedness from experts, philanthropists, and CEOs of AI companies. The Japanese doomsday cult responsible for the 1995 Tokyo sarin attack had money, facilities, scientifically educated members, and a leadership willing to cause mass harm. If any terrorist organization was well positioned to develop a large-scale biological weapon, Aum Shinrikyo was it. Yet its bioweapon program repeatedly failed. If Aum couldn’t succeed in its pursuit of bioweapons, what non-state terrorist organization can?Biological weapons are undoubtedly difficult to develop. Their production and use involve multiple interdependent problems, including acquisition and identification of suitable material, cultivation, scaling, purification, weaponization, and deployment. Aum encountered real scientific barriers. However, the claim that the group “had biological experts yet still failed” overlooks the likelihood that members didn’t have the right expertise for the tasks they were attempting.Beyond technical skills, successful bioweapon development depends on an organization’s ability to apply knowledge effectively. Richard Danzig and colleagues’ research and interviews with former cult members, on which we draw throughout, show how Aum’s structure and dynamics made an already difficult undertaking harder by preventing knowledge from directing key decisions. Secrecy restricted access to relevant expertise, hierarchy placed authority in the hands of people who lacked the expertise to make sound technical judgments, and internal competition distorted incentives, leading projects to move toward deployment without reliable validation.AI has the potential to address both the technical and organizational hurdles that contributed to Aum’s failure, and extremists are already using it. One of us (Juelich) conducted interviews with former members of Boko Haram. The resulting report, which forms the basis of our account here, shows the group’s factions building specialized AI units to route difficult questions, supplement knowledge, circulate guidance, and help members iterate after unsuccessful operations. Against this backdrop, policymakers, security analysts, and biosecurity researchers can no longer treat past failures of well-resourced non-state groups as proof that they will fail again.Expertise in One Topic Is Not Expertise in AllIt’s true that Aum had biological expertise, but that description is also misleading. Seiichi Endo was made head of Aum’s bioweapons program because of his related education, but “biological expertise” isn’t a single skill. Endo studied veterinary medicine and had started, but did not finish, a graduate degree in virology. Aum concentrated primarily on bacterial pathogens and their toxins. While Endo may have had general laboratory knowledge and a conceptual understanding of different pathogens, his background did not include in-depth pathogen-specific knowledge of the agents Aum tried to weaponize. This knowledge gap is especially pertinent when considering Endo’s difficulty in acquiring or engineering a virulent strain of Bacillus anthracis.Beyond the issue of bounds in expertise within one discipline, a biological weapons program isn’t a single scientific task. Complex engineering efforts by violent non-state actors involve multidisciplinary subtasks that require integration, demanding a variety of technical skills that one person is unlikely to hold. Thus, biological weapon programs require interdisciplinary knowledge including microbiology, aerobiology, chemistry, statistics, and mechanical and electrical engineering.Endo’s expertise did not extend to more operational aspects of large-scale production and deployment, either. He delegated key tasks to members with zero biology training, and members improvised equipment even when they lacked the relevant engineering skills, owing to time pressures and fears of detection. When the group considered purchasing a commercial spraying system for deployment, the anticipated two-month delay prompted the cult leader, Asahara, to order members to construct one themselves. As recalled by Asahara’s personal physician, Tomomasa Nakagawa, the resulting sprayer “broke down over and over again,” and these mechanical difficulties made it “spout like a whale” during attempted attacks.Expertise therefore isn’t a binary variable that an organization either possesses or lacks. This issue is not unique to Aum. Al-Qaeda similarly hoped that recruiting a specialist would be “the fastest, safest, and cheapest way” to establish a biological weapons program. However, Abdur Rauf Ahmed, a doctor in microbiology, complained about the scarce resources, personnel, time, and difficulty of obtaining pathogenic strains of Bacillus anthracis. Importantly, he was also“not prepared to do any of the laboratory work himself.”Accessibility of ExpertiseAI complicates the assumption that access to specialist knowledge depends on the recruitment of numerous experts. AI capabilities are broad and increasing, including within cyber and biological applications. Leading general-purpose language models exceed human experts in demonstrating rare, expert biology knowledge, troubleshooting virology protocols, and correcting protocols seeded with deliberate errors.In “How to Defend Against AI-Designed Viruses,” Jason T. Kaelber distinguishes general-purpose “chatbots,” such as Claude or ChatGPT, from specialist tools for biological design. However, this distinction overlooks how these two technologies can work together. Agentic language models can help to design, build, and fine-tune specialist biological AI tools, and can act as interfaces for their direct use. Guided by five natural-language prompts, Claude used 12 specialized design tools to design hundreds of thousands of potential scorpion toxin binders, which it then whittled down to a couple of hundred likely candidates. Experiments then confirmed that 17 of these designs bind in the lab. The lines continue to blur when considering recent models such as Carbon-X, which takes a DNA sequence as input and outputs natural-language predictions of function, or Pinal, which takes a natural-language description of a desired function as input and outputs a protein design. If general-purpose models increasingly build and operate specialist tools, then safeguards on these models are less marginal to biological risk than Kaelber’s distinction suggests.Another common argument is that AI does not yet provide meaningful help in the lab. Limited in both number and size, existing studies rarely show how model performance translates to real-world uplift — and the accelerating pace of model development may have already rendered them outdated. A small pilot study with 10 participants from the Los Alamos National Laboratory saw novices with access to AI succeed more often than those without, but an even smaller study from Anthropic did not find any differences. The U.K. AI Security Institute states that, in internal studies, “novices can succeed at hard wet lab tasks when given access to an LLM.” Whereas in the largest laboratory uplift trial to date, “low completion rates” for both AI and internet-only groups limited the statistical power to reliably distinguish between them. These studies primarily focus on laboratory novices working independently, rather than scientists with related expertise working within an organization. As Kaelber notes, many thousands of people already have these basic skills, and “the infrastructure for recovering a virus exists in many laboratories in the pharmaceutical, biotechnology, education, and diagnostic sectors worldwide.”A group may already have resources and scientifically trained members, but struggle to coordinate them, so AI could provide assistance without replacing every specialist or eliminating the need for hands-on work. OpenAI’s collaborative work demonstrates improved outcomes within established laboratories through iterative, AI-guided workflows involving both human scientists and autonomous laboratory equipment, demonstrating that useful AI contributions don’t only involve turning total novices into experts. Overall, existing laboratory uplift studies provide mixed and incomplete evidence of improvements from AI assistance. They also leave open the question of whether sustained use of AI within an organization can improve the application of expertise across complex technical projects.In response to the potential risks stemming from the biological capabilities of AI, leading developers, including Anthropic and OpenAI, have taken precautionary steps to safeguard models that could meaningfully assist users with a basic technical background in developing known biological or chemical weapons. Some protections, such as those applied to Anthropic’s most capable models, extend to broader dual-use biological research. Safeguards can reduce misuse, but protections vary across providers, and independent testing has identified successful jailbreaks in frontier models. Unauthorized third-party services can also undermine access restrictions and route requests across providers, meaning that an effective safeguard or account ban at one company may not prevent a user from obtaining assistance elsewhere.Open-weight models present a further challenge. Their biological capabilities are increasingly close to those of leading closed-weight models, but users can download and host them independently, and improvements in efficiency are allowing increasingly capable models to run on consumer hardware. Users can weaken or remove built-in safeguards through fine-tuning, and local deployment means use is beyond the developer’s monitoring and control.Therefore, safeguards cannot reliably prevent terrorists from obtaining useful AI assistance, and the group’s access to knowledge extends beyond its members’ qualifications and training.Given the Right Knowledge, What Remains?Even if Aum had collectively possessed all the relevant knowledge, it struggled with many of the “intangible factors” that determine whether members use expertise effectively: how they share knowledge, how they exercise authority, how they win and protect status, and how they diagnose and correct failure.The Costs of SecrecyRestricting knowledge of biological weapons development to a select group of leaders seems sensible, but it also shrinks the pool of people available to carry out the program. For Aum, this limitation led to disruption when leadership was otherwise engaged with the needs of the wider organization, such as battling litigation or spending a few weeks in jail. Restriction of information prevented expertise outside its inner circle from contributing to the development of the program, leaving Endo isolated and poorly positioned to oversee all that it required. War on the Rocks Membership Newsletters. Podcasts. Analysis. Exclusive members-only dispatches, premium shows like The Russia Contingency with Michael Kofman, and expert analysis of the conflicts shaping national security. Military, government, and students 33% off with a .mil or .gov email50% off with a .edu email Choose annual, then sign up with that email address. No code needed. Rank Versus CompetenceAum’s centralized leadership had mixed benefits. It allowed Asahara and his senior followers to launch programs quickly, direct money and personnel toward them, and create divisions of labor. But it also made rank and proximity to Asahara more important than technical judgment and sound decision-making.Endo achieved his position in the pecking order primarily due to his religious and fundraising achievements, rather than his biological knowledge. By contrast, Endo’s competitor, Masami Tsuchiya — who led the eventually successful chemical program — remained junior and exercised comparatively little control despite possessing technical skills that proved consequential to the chemical program.The strict hierarchy therefore lacked reliable ways to ensure that technical competence took precedence over rank, preventing members from halting, delaying, or redirecting decisions that those above imposed, and distorting program direction. This led to scale-up failures, production accidents, and, in one notable case, a truck catching fire during the deployment of sarin, despite more junior members having warned leadership of this exact eventuality.Vying for Power, Hiding FailureAum’s internal status system shaped how members interpreted and reported results. From 1992 to 1994, Endo continued to promote biological agents, while Tsuchiya’s work with chemical agents emerged as a competing program. During an attempted assassination in November 1993, Aum reportedly used both sarin and botulinum because Endo wanted a basis for claiming at least part of the credit if the operation succeeded, indicating that technical choices, personal standing, and competition for leadership approval had become intertwined.In a hierarchy where members vie for power and attention from those in the rungs above, a weapons program could bring funding, authority, and proximity to the leadership. Admitting a project is failing might therefore weaken one’s position and create incentives to conceal problems, rather than find solutions. Under such pressure to perform and progress quickly, Endo’s program repeatedly moved toward production and attempted use before establishing whether its material worked as intended. Again, such characteristics are not unique to Aum. For example, despite the technical skills that the Irish National Liberation Army possessed, “almost constant internal jockeying for position” prevented that talent from producing sustained weapons innovation.Not Learning from FailureIt would be inaccurate to say that Aum never learned from its experiences. The group’s members redesigned fermenters, experimented with different dissemination forms, and eventually shifted attention toward chemical weapons following demonstration of their success. In the face of failure, however, the lack of a reliable process for determining what had failed and why disrupted this evolution.In a revealing incident, one member fell into a production tank of Clostridium botulinum that should have contained lethal botulinum toxin. Although the individual nearly drowned, they otherwise suffered no ill effects, and subsequent inoculation of mice with the culture produced ambiguous results. They nevertheless used the culture material in attacks, but these attempted acts of bioterrorism went unnoticed, largely due to their lack of impact.When the attacks failed, leaders presented a range of possible explanations. They argued that the material wasn’t harmful, the spraying equipment didn’t work correctly, or they had simply misjudged the wind. However, the group failed to resolve any of these possibilities, so their anthrax work followed a very similar pattern: It barreled toward mass production without establishing virulence, accidental inhalation had no impact on those exposed, and repeated malfunction of improvised dissemination equipment led to neighbors alerting police due to “foul odors.”The issues discussed so far culminated in a lack of troubleshooting, iteration, and improvement. Secrecy restricted access to relevant expertise, hierarchy perpetuated unwise decision-making, and competition engendered personal motivation to preserve an ineffective program. A technically faulty system could therefore progress all the way to deployment without anyone pulling the brake.AI as Organizational InfrastructureAum’s organizational struggles are not unique. Many non-state armed groups face internal conflict and rivalry, principal-agent problems, and the competing demands of secrecy and oversight. These issues can lead to a range of negative outcomes from the perspective of the group, from unordered violence to foiled attacks, and can hinder organizational learning to gain new capabilities by constraining the access, flow, and interpretation of relevant information.Warnings about terrorist use of AI have focused predominantly on models as a source of technical knowledge. Less attention has gone to the structures groups build around it. The study on Boko Haram documents how the group is building AI into exactly these structures. Drawing on 57 in-person interviews with 27 former members — including commanders and technical specialists — it shows a group reorganizing itself to adopt a new technology, and building processes for acquiring, interpreting, and circulating relevant knowledge.Boko Haram formed specialized AI teams drawn from senior personnel with technical or operational backgrounds, including bombmakers, computer specialists, and engineers. Operatives of transnational jihadist groups, primarily linked to the Islamic State, trained the units, which sat directly below leadership within the hierarchy. As one former member explained: “They don’t go to war. Their role is to disseminate information.”Although AI access remained concentrated among trusted personnel, guidance circulated through the hierarchy, and commanders could escalate problems to the AI units. “When we have problems we can’t solve ourselves, we run to them,” a former commander stated. The units would query a model, interpret its response, and communicate guidance for implementation, creating a repeatable route for sharing knowledge and iterating on issues.Trust in AI could influence whose judgment prevailed, with another former commander describing human judgment as secondary: “If you don’t get what you need from AI, we use our brain to supplement.” Others described the AI units prompting changes in the intended courses of action by communicating a clear “No, don’t do that!” The position of these units in the hierarchy may therefore have made members more likely to trust and follow technical guidance.Commanders reported feeding accounts and footage of unsuccessful operations to the AI units for analysis and correction. If a trusted problem-solving framework exists within an organization, members may be more willing to surface failures for solutions. Similarly, leaders who believe AI can make difficult problems more navigable may commit more personnel, time, and money to intended approaches, and may accept persistence after failure if they can revise the next attempt, rather than just repeating it.The report on Boko Haram documents these changes and the perceived improvements in capabilities, but the extent of any actual improvement remains unclear. Our inference from Boko Haram’s AI usage to biological risk is organizational, and there is no evidence that Boko Haram is pursuing biological weapons. Failures to bring specialist knowledge into sensible decision-making, circulate guidance effectively, abandon ineffective approaches, and use unsuccessful attempts to guide corrective actions undermined Aum’s program. The channels for guidance, escalation, and feedback that Boko Haram institutionalized could address these same failures in an organization developing a biological program. Of course, the technical advice and experimental validation would need to be specific to biology, but organizational functions extend across technical disciplines. When a group follows reliable guidance, these structures help it use its expertise and resources more effectively, avoid preventable errors, and reduce friction toward success.Significant practical and physical barriers to developing biological weapons remain, and equipment, materials, and practical laboratory skills are still essential. But these requirements do not make AI assistance irrelevant, and some have informational, operational, and organizational components that AI could help actors address, such as identifying ways to evade controls on access to reagents, materials, and pathogens.Aum remains a foundational case for understanding what leads a group to attempt bioterrorism. Its failure, however, is a less reliable guide to what a similarly motivated group could achieve today. Given the increasing biological and operational capabilities of frontier models that could erode historical barriers, policymakers and security agencies can no longer treat the difficulty of biology, or the dysfunction of extremist groups, as a guarantee that attempts will fail. Further, the prior rarity of severe biological attacks does not establish that the frequency of attempts will remain unchanged as constraints shift. Risk assessments should account for how AI changes access to specialist knowledge and the structures through which actors can apply that knowledge.Our analysis strengthens the case for, and adds an organizational dimension to, Kaelber’s call for AI safeguarding and monitoring, as both must account for how terrorist groups are integrating general-purpose AI into decision-making and learning. Security agencies, researchers, and AI companies should work together to investigate how terrorists use AI and update biological risk assessments to anticipate where it could narrow the gaps between intent, capability, and opportunity.Such findings can guide AI developers in strengthening layered safeguards, combining model-level protections with rapid detection and disruption of misuse, fixes for identified jailbreaks, and enforcement of access restrictions across third-party services. Despite ongoing efforts, protections remain uneven across the AI ecosystem. Common standards and rigorous evaluations could ensure that safeguards limit biological misuse while preserving legitimate researchers’ access to beneficial capabilities.For open-weight models, developers should prioritize targeted filtering from the outset of pretraining and invest in tamper-resistant safeguards. They should investigate remaining biological misuse risks and safeguard robustness under adversarial fine-tuning, as recommended by the U.K. AI Security Institute, which would then inform whether, and how, model weights are released.To prevent biological threats, we need defense in depth, with layered safeguards tested against and evolving with how terrorists use AI. Controls on biological materials and a sustained investment in incident preparedness and response remain essential, but we must design these measures for an adversary whose tools can help find ways around them. “If Aum Shinrikyo couldn’t build a bioweapon, then no terrorist group can” was never a strong argument. But it is ever more dangerous to hold onto it today.Eleanor Marshall, Ph.D., is head of applied research at SecureBio. She holds a Ph.D. in virology and leads evaluations of AI model safeguards and high-risk capabilities, focusing on understanding and reducing the risk of biological misuse by malicious actors.Antonia Juelich, Ph.D., is the international security lead at the Cambridge Programme on AI Science & Policy at the University of Cambridge. Her work examines the intersection of international security, political violence, and frontier AI risks, focusing on how terrorist organizations misuse AI.Image: April Finnen via Wikimedia Commons

Original Source

Read the full article at Warontherocks →

KhanList aggregates and links to publicly available news content. We do not host full articles from third-party sources. Always verify important information with original sources.