Is Speculative Decoding's Speedup a Hardware Problem or a Model Problem?

A follow-up/sub-part to Part 3 of the LLM inference internals series. Part 3 built sampling-mode speculative decoding with KV caching on both the draft and verifier sides, and it worked correctly, but the speedup it delivered didn't match expectations set by the paper. This post is the record of chasing that gap: every hypothesis tested, which ones were wrong, and what the real answer turned out to be. Starting Point: It's Slower, Not Faster The first full gamma sweep, on an open-ended, opinion-style prompt, came back like this: Mode Gamma Tokens Tok/s Acceptance Measured Speedup Verifier-only - 250 14.4 - 1.00x Speculative 4 250 12.3 39.2% 0.86x Speculative 7 250 9.5 33.6% 0.66x Slower across the board. The accept/reject math had already been verified correct, with stable, repeatable acceptance rates and no correctness bugs, so this wasn't a bug. It was a real result that needed an explanation. Hypothesis 1: It's MPS Dispatch Overhead The first instinct was to blame the hardware. Apple Silicon's MPS backend pays a large, roughly fixed dispatch cost per model call that doesn't shrink proportionally with model size. That was confirmed via isolated microbenchmarks: the 6x smaller draft model was only about 2.8x faster per single-token call (21 to 28ms versus 60 to 90ms). Two sequential model calls per round, each paying that fixed tax, looked like the obvious explanation. To test this without conflating hardware cost with algorithm cost, I computed a FLOPs-based theoretical speedup using Leviathan et al.'s formula, E[speedup] = (1 - α^(γ+1)) / ((1 - α)(γ·c + 1)), with a cost ratio c derived from each model's architecture (layers, hidden size, GQA heads, FLOPs per token via the standard 2 × params approximation) rather than measured milliseconds. That makes the number hardware-agnostic by construction. Verifier (Qwen2.5-3B-Instruct): 6,171,394,048 FLOPs/token Draft (Qwen2.5-0.5B-Instruct): 987,922,432 FLOPs/token c = 0.1601 Enter fullscreen mode Exit fullscreen mode Plugging in the real measured acceptance rates: γ Acceptance Theoretical (FLOPs-only) Measured (MPS) 4 39.2% 0.99x 0.86x 7 33.6% 0.71x 0.66x 8 19.7% 0.55x 0.55x This was the first wrong turn. Even with zero hardware overhead, the theoretical ceiling never cleared 1.0x. If MPS overhead were the whole story, the theoretical column should have shown a clear, comfortable win being eaten by dispatch cost. Instead it showed that the algorithm itself, on paper, wasn't going to win at this acceptance rate. Hypothesis 1 was only a partial explanation at best. Hypothesis 2: A Bigger Draft Model Would Fix It If acceptance rate was the real ceiling, the obvious next move was a closer-sized draft: Qwen2.5-1.5B instead of 0.5B, on the theory that a closer size means a closer distribution match to the verifier. Checking this on paper first, via AutoConfig with no weights downloaded, gave a worse cost ratio: c = 0.5002, about 2x cheaper than the verifier instead of 6x. Solving the formula for breakeven acceptance rate at each gamma showed the 1.5B candidate would need 69 to 84 percent acceptance just to reach 1.0x, far above anything measured so far. Second wrong turn, ruled out before spending compute on it. A bigger draft looked like trading one problem, low acceptance, for a worse version of the same problem, an unfavorable cost ratio. Then I actually ran it. Real numbers, not just theory: Run γ Acceptance Theoretical Measured 1 6 86.4% 1.18x 1.37x 2 6 19.8% 0.31x 0.48x 3 6 39.5% 0.41x 0.72x 4 6 30.1% 0.36x 0.59x One run cleared 1.0x by a wide margin, the first of the whole investigation. But three others landed back in the 20 to 40 percent range the FLOPs math had already predicted was a losing zone. This wasn't the bigger-draft theory being confirmed. It was acceptance rate itself being far noisier and more content-dependent than a single-number cost-ratio model could capture. The 1.5B draft wasn't reliably better, it was occasionally much better, which pointed at something other than model size driving the variance. There was also a second anomaly in this data. In several rows, measured speedup exceeded the theoretical ceiling, which is supposedly impossible since theoretical was meant to be a zero-overhead upper bound. That turned out to be a real finding, not a bug. FLOPs-based c assumes cost scales with parameter count, but on MPS, dispatch overhead compresses the real cost gap between a 1.5B and 3B model far more than FLOPs alone would predict. The FLOPs-based ceiling is a valid algorithmic upper bound, but it is not a valid hardware upper bound on MPS, a subtlety worth naming rather than glossing over. Hypothesis 3: It's the Prompt The high-variance 1.5B result raised the real question directly: what made that one run hit 86 percent acceptance? The recurring guess was entropy. A prompt with structured, predictable, lower-branching continuations should let a small draft model track a large verifier much more closely than an open-ended, creative one. To test this cleanly, I swept temperature (0.01, 0.3, 0.7) and top_p on the original opinion-style prompt, and acceptance barely moved, staying in the 20 to 45 percent band regardless of sampling settings. Then I swapped the prompt itself: same models, same code, same hyperparameter ranges, but asked for a detailed summary of a structured, factual passage (photosynthesis) instead of an open-ended question, at 500 tokens: Mode Gamma Tok/s Acceptance Measured Speedup Verifier-only - 13.1-14.3 - 1.00x Speculative 4 12.8-15.5 43.6-54.0% 0.95x-1.16x Speculative 6 13.7-15.3 45.0-52.6% 0.99x-1.08x That's the first time changing a single variable moved acceptance meaningfully, and it wasn't temperature or top_p, it was the content itself. Structured, factual text is where this draft and verifier pair actually performs. Open-ended, creative text is where it doesn't. This became the real answer to what causes the acceptance ceiling here: not the models being poorly matched in general, but poorly matched specifically on high-branching-entropy content, a hypothesis Phase 4's entropy correlation study can now test directly instead of guessing at. Hypothesis 4: It's Still Just MPS, Even at High Acceptance With acceptance now reliably in the 45 to 54 percent range and several runs already crossing 1.0x on PyTorch and MPS, the last open question was whether the shortfall from the 1.5 to 2x range reported in the literature was still a hardware story. To check, I ran the same models (GGUF, Q4_K_M quantization) through llama.cpp on the same photosynthesis-style prompt: n_draft = 4 n_predict = 504 n_accept = 391 / 448 drafted -> 87.277% acceptance decoded 504 tokens in 10.581s -> 47.6 tok/s (speculative) verifier-only baseline (same prompt, same runtime): 50.0 tok/s Speedup: 47.6 / 50.0 ≈ 0.95x Enter fullscreen mode Exit fullscreen mode llama.cpp's Metal backend is genuinely faster in absolute terms, roughly 3x the raw tok/s of the PyTorch and MPS setup on both the baseline and the speculative run, confirming the earlier microbenchmark finding that MPS pays real dispatch overhead PyTorch doesn't fully hide. But at 87.3 percent acceptance, well above anything achieved on PyTorch, llama.cpp's own speedup was still around 0.95x, not the 1.5 to 2x hoped for. Fourth wrong turn. If MPS and PyTorch overhead were the primary blocker, a near-zero-overhead runtime at excellent acceptance should have shown a clear win. It didn't. llama.cpp printed something that pointed at the real remaining factor: 35.7 percent of total wall-clock was unaccounted time, not verifier compute, not draft compute, not sampling, but round-trip and orchestration cost between the draft and verify phases. That's consistent with both implementations, this repo's and llama.cpp's CLI example, running draft and verify strictly sequentially, round after round, rather than pipelining verification of round N with drafting of round N+1. What Actually Explains the Gap Putting all four hypotheses together, in order of how much each one turned out to matter: Per-round orchestration overhead, not per-call dispatch cost, is the main remaining bottleneck at good acceptance rates, confirmed by llama.cpp hitting the same 0.95x ceiling at 87 percent acceptance despite near-zero dispatch overhead. Content and entropy drive acceptance rate far more than temperature, top_p, or draft model size, the single biggest, most reproducible lever found in this whole investigation. MPS dispatch overhead is real but secondary. It explains part of the PyTorch versus llama.cpp absolute speed gap, but not why even the fast runtime falls short of the literature's numbers. A bigger draft model is not a reliable fix. It's occasionally much better, but with high enough variance that it isn't a dependable lever on its own. Why the Paper's Numbers Are Higher Leviathan et al.'s headline results didn't come from one trick. They stack several advantages this setup doesn't have: A much larger size gap (T5-XXL to T5-small, roughly 180x parameters) versus this pairing's roughly 6x. A bigger absolute verifier cost makes batching gamma plus 1 tokens into one pass save proportionally more wall-clock time. Drafts trained to align with the verifier, not independently instruction-tuned same-family siblings. This is very likely the single largest acceptance-rate lever available, and this repo's Qwen2.5-0.5B and 3B pair has none of it. Pipelined, overlapped execution between draft and verify stages, rather than strict sequential round-trips, directly addressing the orchestration overhead this investigation found via llama.cpp's unaccounted time. TPU and datacenter-GPU dispatch overhead, lower even than llama.cpp's Metal backend, shrinking the fixed per-round tax further. None of these is a bug in this implementation. They're conditions the paper's results depended on that this project, by design (local, from scratch, consumer hardware, off-the-shelf same-family models), doesn't have. Limitations Draft and verifier size ratio and training. A 6x parameter gap versus the paper's roughly 180x, and no distillation. The draft was never trained to mimic the verifier, which is very likely the dominant reason acceptance tops out where it does even on favorable content. Sequential execution only. Both this implementation and the llama.cpp comparison run draft and verify strictly in sequence. No round-pipelining was attempted, so the roughly 35.7 percent orchestration overhead measured here isn't necessarily a hard floor, just what this simple execution model costs. Hardware and runtime. Results span PyTorch and MPS, and llama.cpp and Metal, on Apple Silicon only. No datacenter GPU or TPU data point exists to confirm how much of the remaining gap is Apple Silicon specific versus fundamental to unpipelined two-model decoding. Sample size. Acceptance-rate variance was large enough (19.8 to 86.4 percent on the same 1.5B and 3B pair, same gamma) that single-prompt, few-run comparisons should be read as suggestive, not conclusive. The entropy and content hypothesis needs Phase 4's larger prompt set to confirm. FLOPs approximation. The 2 × params estimate doesn't model attention's quadratic term in sequence length, and the theoretical formula assumes no fixed per-round overhead. Both simplifications that this investigation's own data (measured beating theoretical, llama.cpp's unaccounted time) shows don't fully hold in practice. Series: Part 3, Speculative Decoding: Sampling-Mode Accept/Reject, leads into this post, which leads into Part 4, The Empirical Study (coming)