I trained my local LLM on its own failures, and now it learns from every mistake

Published Sep 17, 2026, 12:30 PM EDT Maker, meme-r, and unabashed geek, Joe has been writing about technology since starting his career in 2018 at KnowTechie. He's covered everything from Apple to apps and crowdfunding and loves getting to the bottom of complicated topics. In that time, he's also written for SlashGear and numerous corporate clients before finding his home at XDA in the spring of 2023. He was the kid who took apart every toy to see how it worked, even if it didn't exactly go back together afterward. That's given him a solid background for explaining how complex systems work together, and he promises he's gotten better at the putting things back together stage since then. I've been using AI models since they first became publicly available. Well, that's not exactly true. Decades ago, I was at university for AI and Psychology, and while I found the study of the brain more interesting than learning to code in C+, my curiosity about creating computers that can think spans back even further. The local LLMs I use now are a different type of learning than what was in the field back then, but they're no less fascinating to me. I've been using them for all kinds of daily tasks, but they're not without their own issues, and one is that they don't learn beyond their training. If they get something wrong and are corrected, that usually doesn't survive outside the context of that individual chat. So I built a small app with Codex's help that stores checked corrections, so that the model can add that data when it needs it. It's not changing the model, but giving it persistent memory gives it a FAQ to rely on for future reference. My local LLM now has a notebook for corrections I'm still responsible for the answers, however The vibe-coded app is called Mistake Memory, and it's simple at its core. Some Python code to handle requests, SQLite to store the corrections and other context, Lemonade serving a small Qwen model, and a web UI to give me an easy interface to work with. It uses things I already had installed: Python's standard library and Lemonade's OpenAI-compatible API, so I don't have to worry about dependency creep. The UI has four sections: Chat, Corrections, Compare, and Export. At some point, I'll rewrite things as a skill to use in my agentic coding tools, so I don't have to open the UI to paste in questions and corrections, but this works well for a proof of concept. Each question in the Chat tab starts a fresh conversation to avoid earlier context from ruining the data, and there's a toggle to use previously saved corrections. If the model returns an incorrect answer, I can go to the corrections tab and save the original question, failed response, a verified replacement, and a rule set to hopefully avoid getting wrong answers later. That gives the model all the context it needs to figure out future scenarios without me having to nudge it. And yes, I can edit corrections if they still result in wrong answers, with a continual learning process. It's a manual, low-flow version of the training that models go through before they get to me. I did have one issue along the way: Qwen used up all its token budget on thinking and didn't return an answer a few times. Disabling thinking for the quick tests gave me results to work with, and I can fix the token limit later. Lemonade Lemonade is a quick way to self-host LLM models for use. Finding Qwen's mistakes was the easy part Fixing them took longer The tasks I started with were drawn from things I've been using local models to accomplish. Things like formatting filenames, parsing CSV data, sorting entries, and working through device logs. These details matter if you want to trust a local LLM to handle your downloads folder before letting it handle proprietary data. I ran 22 questions without memory, and selected ten that returned incorrect answers. It used temperature 0.6, seed 42, a 512-token response limit, and thinking disabled. These were deliberately selected failures under those settings, so they don't tell us Qwen's overall accuracy. They do give me something concrete to work on. One filename task asked Qwen to turn NAS Backup.v2 [FINAL].tar.gz into nas-backup-v2-final.tar.gz. It got close, but dropped the "nas" part entirely. Just as well this was training and not touching real files. Elsewhere, it added quantities from a CSV and somehow added two and three to get eight. The corrections worked even after an app restart, which is promising, but it doesn't give the local LLM any better reasoning skills. It just means it knows where to look for some types of answers. Answers that I can verify using the Compare tab, which runs questions with memory on and off, and shows the results to separate failures from unseen variations and control questions. Any exact answers can be graded autonomously, but anything more creative needs the human touch to verify the fixes worked. Now I have a starting point for fine-tuning All models go through this but on a larger scale The Export tab turns my corrections into prompt-and-answer pairs for a training dataset. Failed answers are saved in a separate file, and the topic is attached to each one so that context clues are preserved. Any conflicting corrections are removed from active use until I resolve them, and any questions reserved for unseen tests or controls are excluded as well. That file could be used to train a LoRA adapter, a small set of learned adjustments applied to the local model. Over time, my version of the local LLM will get smarter and more personalized, giving me better answers tailored to my needs. My local LLM isn't much smarter yet but it's a start As of right now, my local LLM is going back to school. Every failed answer becomes a teaching moment for both the model and me, and it's fascinating to see how things progress with updated data. I'm sure I won't be able to out-train the open-source models because I don't have the resources, but what I can do is build a model trained on data that doesn't exist anywhere outside my walls, for the tasks I need it for.