Mathematical Theories Could Be the Key to Explainable AI Systems

Amid growing cybersecurity concerns about AI systems, a consensus in the AI market remains that these systems are largely black boxes and that explainability and governance should continue to be priorities for enterprises.Incidents such as AI agents from frontier labs escaping their sandbox environments and attacking external IT infrastructure indicate there is still much to do with explainability and governance in AI. For businesses, the incidents also mean that they can’t fully trust models on their own and that caution is warranted.For enterprise AI startup Kodamai, one answer to the relative lack of explainability in current AI systems is for enterprises to turn to platforms that apply mathematically grounded theories to their models.“Once you build AI deeply rooted in math, you cannot bias it, you cannot manipulate it, you cannot hack it,” said Maha Achour, Kodamai’s CEO and founder, on the Targeting AI podcast. “Governance is built in, mathematically rooted in our platform.”Related:Alibaba Sells More Shares to Raise $10.2B to Spend on AIKodamai uses advanced math, such as category theory and type theory, to determine that its AI agents are performing their jobs correctly. In category theory, the focus is on how objects relate to one another. On a platform like Kodamai, math is key to ensuring that the relationship and meaning of the data, as it is used among different agents, are never lost.Meanwhile, type theory assigns each data item or system a type and checks that every AI agent manages data correctly. Category theory aims to ensure there is no miscommunication between agents and that the meaning of the data remains intact. Kodamai also uses neuro-symbolic AI to combine its mathematical theories with the pattern-matching of AI systems.However, Kodamai is not focused on building new LLMs; instead, it concentrates on grounding existing LLMs using its math-first approach. Moreover, Achour is convinced there is still a need for humans.“Humans will always be in control when interacting with these very advanced AI platforms,” Achour said. “So, this embedded built-in governance and the correctness that enables these agents to operate and delegate tasks across various departments is important.”About the AuthorsNews Writer, AI BusinessEsther Shittu has covered AI technologies and industry trends since 2021. As co-host of the Targeting AI podcast, she talks with experts, thought leaders and practitioners exploring critical AI developments. Before AI Business, she wrote for SearchEnterpriseAI, the New York Daily News, Bklyner and the Brooklyn Daily Eagle. When she's not diving deep into the world of AI, she spends her time on passion projects and raising her three daughters.Senior News Director, AI BusinessShaun Sutner, a journalist with more than 25 years of daily newspaper experience and 12 years at Informa TechTarget as an editor and writer, directs news coverage for AI Business. He was previously a senior news and features writer covering health IT and HR software at TechTarget and a senior news director overseeing coverage of AI, business analytics, data management and government tech regulation.Sutner's newspaper career included investigative reporting and covering the Massachusetts State House and politics for the Worcester Telegram & Gazette. He has written about snow sports as a T&G columnist and correspondent for 21 years. Sutner's interests also include tennis, standup paddleboarding, cooking and popular music.