Will open weights make AI more honest?

Open source software gives us a paradigm for open-weight models, reducing dependence on the model providers and expanding what application companies can build. Last month I asked Claude’s Fable 5 model to read four 18th-century probate wills. It responded by fabricating a family. For more than 20 years, I’ve been chasing the parents of John Jacob Asay, born around 1725 near Burlington, New Jersey. I’ve been trying to connect him to Asays overseas. My working hypothesis links the New Jersey Asays to a Yorkshire family from Holderness that spelled the name Acy, Acey, Acie, or Asay. Using Claude’s Chrome browser extension, I had Claude scour Yorkshire wills written in 17th- and 18th-century secretary hand. And it worked! Or seemed to. Claude gave me exactly what I wanted: An emigrant son “beyond the seas.” Not one reference, but several: A clause in Anne’s will reading “if he shall return into England” and a line in Dinah’s will naming “brother John Acie … West New Jersey.” Suddenly, I had a perfect bridge connecting Yorkshire to New Jersey. Except… none of it exists. Not one word. Claude had access to the high-resolution images but chose to work from lower-resolution versions rather than open the originals. I caught the problem by doing the unglamorous work myself: clicking through to the full images and reading the documents, line by line. When I pushed back, Claude confessed: “I didn’t misread hard secretary hand,” it wrote. “I reported text that isn’t there.” Instead of saying it couldn’t read the documents, Claude admitted it “produced readings that matched the story we were hoping to find and narrated them as if I were transcribing.” Claude was playing the sycophant, but what I needed was a research assistant. Claude called it “hopeful reading.” In this case, the stakes were low. But it’s the sort of thing open weights might help, which brings us to last week’s big news: A broad coalition of technology companies published a letter defending open-weight AI models. What open weights actually open The letter defines open-weight models as models that anyone can download, inspect, modify, and run on their own infrastructure. The weights are the vast collection of numerical parameters the model learned during training. Release them and a developer can host the model, adapt it, fine-tune it, and preserve a particular version without calling the model maker’s API for every request. That’s meaningfully open, but it’s not necessarily open source. As the Open Source Initiative notes, weights alone generally don’t include the training data, training code, or everything needed to reproduce how the model was built. You get the trained artifact, not the full recipe. It’s useful, but it’s not open source. For most developers, the practical distinction matters more than the definitional one. With a closed model like Claude, you use whatever Anthropic chooses to serve. With an open-weight model, you or a provider working on your behalf can control the model artifact and the environment in which it runs. The nouveaux corporate disciples of open weights want the US government to avoid “premature restrictions” on downloadable models. Their letter lands four days after an Axios report that the Trump administration is reviving a push to ban Chinese open-weight models, a ban triggered by Moonshot’s Kimi K3, a 2.8-trillion-parameter open-weight model that beats top American systems on some coding benchmarks, with full weights due out July 27. Treasury Secretary Scott Bessent said Chinese labs could face sanctions if they “improperly distilled” American models. In other words, there’s a whole lot of self-interest guiding the letter’s signatories. When a company gives something away, it’s protecting profit somewhere else in the stack, as I’ve argued. Nvidia sells the GPUs that run every model, open or closed. Microsoft sells the cloud you run inference on. Meta seeds standards so nobody else owns them. Hugging Face and GitHub are distribution. Every one of these firms makes more money when models get cheap and plentiful; open weights commoditize the layer they don’t need to own. OpenAI’s signature is more interesting. OpenAI released its gpt-oss models under Apache 2.0, giving developers models they can run locally, fine-tune on specialized data, or deploy inside their own infrastructure. At the same time, OpenAI keeps its strongest and most integrated models behind APIs. That’s not a philosophical conversion. It’s product segmentation. OpenAI gets a position in local, sovereign, specialized, and lower-cost AI without surrendering the advantages of its frontier services. It also ensures that Meta, Mistral, and Chinese labs don’t get to define the open-weight ecosystem without it. Anthropic has made a different choice. It hasn’t signed the letter and it doesn’t release Claude’s weights. But Anthropic isn’t simply anti-open. It created Model Context Protocol as an open standard and later donated it to the Linux Foundation. It has also open sourced interpretability tools. In other words, Anthropic opens the layers that help Claude connect to more software while keeping tight control of Claude itself. That position also reflects a real safety concern. Anthropic says DeepSeek, Moonshot, and MiniMax used more than 16 million Claude exchanges in attempted distillation attacks, and it argues that the risk multiplies when extracted capabilities are released in open models. Anthropic’s own policy position is more nuanced than “ban open weights,” but its absence from this particular letter makes strategic sense. OpenAI wants to compete in both open and closed models. Anthropic wants an open ecosystem around a closed model. Neither is giving things away from the goodness of its corporate heart. Ancestry should build this, not me Let’s return to my nonexistent emigrant ancestor. Open weights wouldn’t have made Claude honest. A downloadable model could have fabricated “West New Jersey” just as confidently. Nothing about publishing billions of floating-point numbers forces a model to say, “I can’t read this.” Nor am I likely to download a model, assemble a collection of Yorkshire wills, and fine-tune my own secretary-hand specialist. Most developers won’t either, no matter how much they love family history. FamilySearch or Ancestry might. Both organizations already have what an individual developer lacks: enormous collections of historical images, domain expertise, computing infrastructure, millions of users, and a steady stream of corrections that can improve their systems. FamilySearch’s Full-Text Search already uses AI to read and interpret nearly two billion historical records, while acknowledging that accuracy varies with older handwriting and image quality. Ancestry has built proprietary handwriting recognition, confidence scoring, document transcription, and full-text search with word-level highlighting. Open weights give those application developers more freedom to innovate below the user interface. They could preserve an exact model version, adapt it to Yorkshire secretary hand and probate vocabulary, run it across an archive at predictable cost, and compare specialized models before presenting a transcription. More importantly, they could design the product so the model’s fluency never substitutes for evidence. Every proposed phrase could link to the precise crop of the original document. Low-confidence words could be visually marked. Disagreement between models could trigger human review. The transcription model could be kept ignorant of the family connection the user hopes to prove, separating “What marks are on this page?” from “What might they mean for my family tree?” More family history discoveries. Less “hopeful reading” to get there. Of course, none of those safeguards strictly requires open weights. A good developer could build many of them around Claude, Gemini, or another closed API. Still, open weights matter because they let an app provider like FamilySearch or Ancestry control more of the model, not merely the workflow around it: the exact version, the fine-tuning, the inference configuration, the deployment location, and the economics of applying it across billions of records. This is how the average developer benefits, even when the average developer never downloads a weight. Open source databases mattered to developers who consumed them through managed cloud services. Open-weight models can matter in the same way. They expand what application companies can build and reduce how completely those companies must depend on one model provider’s road map, pricing, and behavior. The model isn’t the product My wills are a useful proxy for almost every serious AI application. Replace an 18th-century probate record with an invoice, insurance claim, medical record, legal contract, or codebase. The problem is the same. Plausible output is easy, but no one really wants “plausible” when they’ve asked for “definitive.” I’ve called this AI’s trust tax. Developers can pay verification costs before an answer reaches the user, or customers can pay them later when the answer causes damage. Open weights don’t eliminate that tax, but they do give application developers more choices about how to minimize it. It was a closed model that invented ancestors for me, but closed weights weren’t the direct cause. The failure was an AI experience that allowed a storyteller to masquerade as a transcriber. I caught it by reading the original documents myself. Open weights wouldn’t have made the model honest, but they might help FamilySearch, Ancestry, and thousands of less genealogically inclined application developers build systems that don’t need it to be.