I ran 252 questions through three local LLMs, and a messy reasoning trace says nothing about the answer

Watching your local model think is one of the quickest ways to lose confidence in local LLMs in general. Even for a simple request, the opening thinking block can look like an internally inconsistent ramble, with a constant self-directed back-and-forth over what you actually wanted. Once it hits the third or fourth attempt, you might feel the need to cancel it, tweak your prompt, and start again. But that's not what you should do. I spent a few days putting that instinct to the test, running 252 questions across three models I have to hand: GLM 5.3 Flash on an ExLlamaV3 quant, served through vLLM on a DGX Spark cluster; Qwen 3.8 Flash Next on NVFP4 with FP8 n-grams paged off an SSD on the Lenovo ThinkStation PGX; and VibeThinker 3B, a small maths-tuned model, in LM Studio on my MacBook. I ran each question two or three times depending on the set, and scored every reasoning trace for how often the model visibly contradicted itself using phrases like "wait," "actually," "hmm," "that's wrong," and "no." Across all 252 runs, models that visibly changed their minds more often were not more likely to be wrong. In the most extreme case, VibeThinker reversed itself 40 times across 4,317 words and still gave me the right answer. The result was consistent across all three models and every difficulty level I tested, and it lines up with a much wider body of research suggesting that a messy reasoning trace is not the same thing as bad reasoning. OpenAI hides its traces so nobody reacts to them Also to prevent distillation, of course I'd assumed for a long time that the reason frontier labs hide their reasoning traces was mainly to protect their advantage, and that's still part of it. Jakub Pachocki, OpenAI's chief scientist, gave another reason in an essay published on September 6th, 2026. He said o1-preview was "deliberately designed... to hide the chain of thought, to protect it from supervision pressure in the long term." A footnote acknowledges that preventing distillation was "a secondary reason", while "maintaining CoT monitorability has explicitly been the bigger priority." A user-visible trace will eventually be optimized to look better, but an optimized trace isn't an honest one. OpenAI's own research says the training process teaches a model that its chain of thought is "a private space where it can think whatever it wants and not be punished," and that training searches for the chains "that are the most instrumentally useful to the agent in getting a high reward final answer." That's different from saying the trace is the cleanest or most accurate account of how a model got there. That instinct you have when a model starts flailing, where you wish it would take a cleaner route to the answer, is exactly the sort of pressure OpenAI wanted to keep away from the reasoning process. DeepSeek demonstrates the same thing from another direction. R1-Zero learned to reason well from pure reinforcement learning, but the paper describes its traces as suffering from "poor readability, and language mixing" and being "often not suitable for reading". For R1, DeepSeek added cold-start data with a readable pattern and filtered out anything that wasn't reader-friendly. The reasoning ability already existed. Legibility came afterwards. Leaked frontier-model traces often look much closer to recent local models than the polished summaries users normally see. In one leaked GPT-5.6-Sol thinking trace, it writes "Realizing tool name is functions.exec absent in listed namespace?" Another says "Noticing repeated tool misuse due UI autocomplete?" Articles disappear, verbs become nouns, and it's obvious from the style that nobody is meant to read it. Across 252 runs, back and forths tracked difficulty rather than errors The worst-looking trace I got was correct The worst-looking trace I saw in all of my testing came from VibeThinker 3B, flip-flopping 40 times across 4,137 words of reasoning. It was also correct. The question was to reverse the word "reasoning" and give the fourth letter. It started by getting the reversal wrong twice, producing "gninson e r" and then "gninosear" before eventually converging on "gninosaer", with a "wait careful" and an "actually" in the opening few lines. It then corrected itself another thirty-odd times, worried at length about whether it was formatting the final line properly, and answered n. Which is right. Watching the trace made it tempting to stop the model almost immediately, because it looked like there was no chance it was going to recover. But all three models behaved similarly; across the entire test, 249 of 252 runs were correct. On my "middle" difficulty set, 24% of GLM's correct answers contained an explicit self-reversal, as did 19% of Qwen's. On a harder set built specifically to break them, with six-digit exact multiplication, 7^222 mod 1000, MISSISSIPPI permutations with no two S's adjacent, and the trailing zeros of 617 factorial, those figures rose to 29% and 30%. Only three runs in the entire set failed, and two of those contained exactly one reversal each. Plenty of correct runs did too. A reasoning trace arguing with itself told me very little about whether the final answer would be right. Model size showed up much more clearly in how much reasoning was required. VibeThinker got 41 of 42 questions right on the middle set and 27 of 28 on the hard one, roughly matching models many times its size. Its reasoning traces were enormous, though. On the middle set, 46% of its runs contained a total pivot in reasoning, compared with 24% for GLM 5.3 Flash. On the hard set, that climbed to 89%. Its character-counting traces ran to as much as 2,526 words and fifteen reversals, and every one of them was right. OpenAI calls part of this tradeoff the monitorability tax. Its research found that a smaller model at higher reasoning effort can match a bigger one at low effort while being easier to monitor, with the cost appearing as more computation. VibeThinker showed the same basic pattern here: it could produce GLM-grade answers, but sometimes needed roughly six times as many reasoning tokens to do it. I split my traces into two broad camps: chaotic ones with obvious internal back-and-forth, and ones that went straight to the point. GLM's straight traces averaged 75 words against 210 for chaotic ones. Qwen came in at 76 and 243. These are two models from different companies, quantized entirely differently, yet the split was almost identical. One question pushed both of the larger models every single time. Given packs of 6, 9, and 20 widgets and asked for the largest number you can't buy exactly, both produced traces full of abandoned clauses and corrections. GLM ran through sequences like "26 (6+20), 27, 28 (9+9+... wait 28 = 20+8? no. 28 = 9+9+... 9+9=18, +6=24, no. 28 = 20+... 8 not representable. Hmm, 28: 6a+9b+20c" for hundreds of words before arriving at a clean, correct answer of 43. Famous riddles like bat and ball, the lily pads, and Sally's sisters behaved completely differently. None produced a reversal in 42 runs. The models appeared to recognise the problem almost immediately and then verify the familiar answer. The answer is usually right before the model evaluates other options It can find the answer very early on Beyond the Commitment Boundary, a paper published in June, describes the point where a model stops producing tentative guesses and locks onto a stable answer. That point tends to arrive inside a single reasoning step, well before the trace ends. Early-exiting there cut chain-of-thought length by up to 55% with negligible effect on results. I checked that against my own traces by finding where the correct answer first appeared as a fraction of the way through. Across 77 of them, the answer had already been written down before the trace was one-third complete 47% of the time. The packs of 6, 9, and 20 problem is a good example. Qwen 3.8 Flash Next found the right answer 19% of the way into a 787-word reasoning trace. One GLM run found it 9% into 459 words, while Qwen got there just 6% into a 424-word trace. Most of what followed was not searching for the answer at all. It was verification. In one GLM run, the model spent that second phase testing values from 26 upwards, explicitly flagging 28, 31, 34, and 37 as impossible, then checking that 44 through 49 were all reachable before finally committing to 43. All three of its visible reversals happened during this verification stretch, after it had already found the right answer. The model recognised the Frobenius coin structure quickly (fun fact: sometimes referred to as the Chicken McNugget Theorem), but still spent most of its reasoning verifying the result. That is an important distinction when you're watching one of these traces live. A large amount of what looks like indecision can happen after the useful answer has already appeared. There's no algorithm underneath The trace has to be a mess If I ask GLM 5.3 Flash to multiply 48,291 by 73,846, it never simply "multiplies" two large numbers together. It breaks the problem into 48,291 times 70,000, then 3,000, 800, 40, and 6, accumulating partial sums as it goes. Once it reaches 3,566,097,186, it does the whole thing again through a different route, calculating 73,846 times 48,000 plus 73,846 times 291, then turning the second part into multiplication by 300 minus multiplication by 9. Two independent decompositions produce the same result, and only then does it commit. These traces are essentially working memory. The model can't keep every intermediate value available indefinitely without writing it down somewhere, so the thinking block isn't simply a description of the computation. It is the computation. Research on algorithmic reasoning makes that distinction especially interesting. In one paper, researchers got transformers solving algorithmic problems while filling the reasoning space with meaningless dots rather than meaningful intermediate explanations. What the extra tokens seemed to provide was serial compute, not necessarily conventional human-readable reasoning. Other work has shown that even noisy or imperfect traces can still support correct solutions. The individual arithmetic primitives are learned too. Research studying how models perform arithmetic has found that a sparse set of multilayer perceptron neurons carries many human-readable arithmetic patterns. There is no neat symbolic long-multiplication routine sitting underneath the model. Instead, many overlapping learned patterns cover enough of the input space to make individual small steps reliable. Ablating those neurons can break arithmetic capability entirely. An unreliable primitive plus a scratchpad can still produce a reliable answer if the model decomposes the problem until each step is small enough, then checks it through another route. "Let's do methods... check with alternate" looks messy when you read it as prose, but it makes much more sense when you read it as error correction. That's also why telling a model to be concise is a worse idea than it sounds. There's a difference between a model trained to reason in compressed form, which learns to represent the same intermediate ideas in fewer generated tokens, and an existing model forced into brevity, which simply has fewer reasoning steps available and less room to explore or verify. The first keeps the ideas and drops the tokens. The second can drop both. Tencent's published work on underthinking found that models switching too quickly between reasoning approaches correlate strongly with wrong answers. Forcing brevity can manufacture the same behaviour by denying the model enough room to finish an approach before moving on. There's also some evidence that the visible style of a trace can change without accuracy changing at all. In one small independent Qwen3 experiment using MATH-500, the model scored 97% in both English and Chinese, but used only 61% as many tokens in Chinese. The English traces were full of phrases like "wait, but" and "let me check", while the Chinese ones were much more direct. The surface appearance of hesitation changed dramatically, while the result did not. A model failing often has shorter traces It recalls instead of verifies Qwen 3.8 Flash Next missed the digit sum of 2^100, which is 115. What's interesting is that its reasoning trace was only 123 words long and contained exactly one "wait". At the difficult part of the calculation, it stopped deriving and switched to recall instead. It then wrote: "Known Project Euler #16: sum of digits of 2^100 = 136. Yes answer 136." The second time I ran it, Qwen got the answer right. That trace was 213 words and also contained exactly one "wait", but this time, when it reached the hard part, it kept going. It rebuilt 2^100 from 2^10 and 2^20 upwards and then summed the digits carefully. What separated the two runs was not how often the model doubted itself, instead, it was whether it derived the answer or gave up and recalled one. The recall, unfortunately, was wrong. Project Euler problem 16 asks for the digit sum of 2^1000, and that answer is 1366, not 136. When Qwen stopped doing the arithmetic, its trace became shorter and much more confident. VibeThinker's failures showed the opposite side of the same problem. In one run, it decided that six 4TB drives in RAID 5 gave roughly 9.3TB usable because it never applied the N-1 rule at all. It's a maths-tuned model, but it lacks a lot of real-world factual knowledge. VibeThinker's other failure wasn't really a wrong answer. On the same 2^100 question, it started thinking and simply never finished. After 5,127 words, it hit the 16,384-token cap while doing base-1000 carry arithmetic to construct 2^100 from scratch. Its final line was "c9: r9 = 13,291 mod 1000 = 291? Wait compute: 13*1000=1". Up until the cutoff, the reasoning looked like it was still going somewhere useful. The 40-reversal trace that got an answer right and the 5,127-word trace that never got to finish look remarkably similar for their first couple of thousand words. The important difference isn't that one looks sane and the other doesn't. It's that one was allowed to complete the computation. So if you're about to kill a local model because its thinking looks wrong, you're making that call on the strength of something OpenAI keeps out of sight because optimizing it for human approval could make it less useful. My advice? Let it run.