I'm running a 284-billion-parameter model across two machines, and it finally matches the cloud

Published Aug 15, 2026, 12:30 PM EDT I’m Adam Conway, an Irish technology fanatic with a BSc in Computer Science and I'm XDA’s Lead Technical Editor. My Bachelor’s thesis was conducted on the viability of benchmarking the non-functional elements of Android apps and smartphones such as performance, and I’ve been working in the tech industry in some way or another since 2017. In my spare time, you’ll probably find me playing Counter-Strike or VALORANT, and you can reach out to me at adam@xda-developers.com, on Twitter as @AdamConwayIE, on Instagram as AdamConwayIE, or u/AdamConwayIE on Reddit. Sign in to your XDA account A few weeks ago I ran DeepSeek V4 Flash on a single Lenovo ThinkStation PGX, and it worked, even if it had some pretty big caveats. The ds4 engine is a cleverly designed C project built for a single model family, and it hit 10 to 14 tokens per second with a million-token context window. But that's still slow enough that you sit and watch it, and to fit 284 billion parameters into 128 GB at all, ds4 has to quantize the routed experts down to 2 bits. So the model was both slow and measurably worse than what DeepSeek hosts. Both come out of the same constraints: 128GB of VRAM and 273 GB/s of memory bandwidth. Nvidia recently sent me two DGX Spark machines for testing, which conveniently deals with both caveats at once. Two boxes can pool their memory together to hold the model weights, so the it no longer has to be squeezed to 2 bits, and each box only has to read half of them on every token, which is where most of the extra speed comes from. Pooling them together changes things significantly: the pair delivers 36 to 41 tokens per second on the vanilla model, and bursts to 76 on code with the DSpark speculative-decode checkpoint, while also letting me run the model at native FP8. Meanwhile, prompt processing goes from something you have to to complete to something you never notice. Even better, the whole lot sits on a desk, never pulls more than 272W, and never sends a token anywhere. It's not a plug-and-play setup; you'll need to do a lot of research to identify what works, and I discovered gaps in community-shared guides that took some untangling. Still, the result is a complete local inference stack that's incredibly powerful, and virtually identical to the hosted API from DeepSeek. Nvidia DGX Spark Two Sparks tripled the throughput There are a few reasons why Each DGX Spark has an Nvidia GB10 SoC (a very similar chip to the recently-announced RTX Spark SoC), and each has a Blackwell GPU and 128 GB of unified memory. Both are connected by ConnectX-7, which allows up to 200 Gbps communication and RDMA. You split the model across both nodes using tensor parallelism, so each Spark holds half the parameters and the two communicate on every token through the fabric. What that gets you is 37 to 41 tokens per second on standard writing on a single stream, which is roughly three times what the PGX managed. That's also with a move from ds4 to vLLM, from 2-bit weights to FP8, and the addition of speculative decoding. In fact, if memory bandwidth alone had limited the single box, its much smaller 2-bit weights should have made it faster, not slower. The part that really is down to the second machine is weight sharding. Each node streams only its own half of every layer out of local memory, so the two 273 GB/s buses are working at the same time instead of one sitting idle while the other reads, and since decode is limited by how quickly you can pull weights in, halving the bytes per node accounts for most of the gain. Where those bytes actually go surprised me, though. The routed experts are stored at 4 bits and only six of the model's 256 fire on any given token, so they end up cheaper to read than the attention layers, which run at full precision and are active every single time. It doesn't scale to double though, and the reason is the model rather than the hardware. V4 Flash uses Multi-Head Latent Attention (MLA) with a single key-value head, which means the KV cache is replicated on both nodes instead of split, so both machines are reading all of it on every token. On a 2,048-token benchmark prompt that barely shows up, but when using a 100,000 or higher token context people actually run agents at, it's a meaningful amount of memory traffic getting nothing whatsoever from the second box. Prefill is where you get a big upgrade, and my benchmarks put it anywhere from 1300 tokens per second to 1900 tokens per second. That's compared to the few hundred a single-node ds4 setup manages. With a large code repository, it's a game changer. The quantization change is the big one, though; 2-bit is about as tight as quantization gets before it starts rambling, and I could sometimes see it in large contexts. Running at native FP8 with none of the extra down-quantization means that I'm able to run the model that DeepSeek actually serves. DSpark nearly doubles your peak speed on code Speculative decoding does a great job The base stack is already respectable at 36 to 41 tokens per second, but there's an even faster way to run the same model. DeepSeek published a separate checkpoint called DeepSeek-V4-Flash-DSpark with dedicated speculative decoding layers baked in, and it runs on a different container image that fixes a warmup bug the stock vLLM build has on GB10 hardware. There are three things contributing to the speedup: the checkpoint has DSpark speculative layers which draft up to five tokens per forward pass, and the container image warms the sparse-MLA path that stock vLLM skips on sm_121. Without that fix, the kernels JIT-compile mid-generation. Finally, the KV cache moves to NVFP4, fitting roughly 1.27 million tokens into about 8.5 GiB per rank, so the full context window costs far less memory than a typical FP8 deployment does. NVFP4 stores the KV cache at 4 bits rather than 8, but it isn't comparable to conventional software integer formats people normally use for KV cache quantization. Formats like q4_0 use uniform quantization across values spanning orders of magnitude, wheras NVFP4 retains an exponent and dequantizes directly on Blackwell's tensor cores. Even just for speed that's important, with one test measuring a 37% decode penalty for q4_0 at 110K context. This is because software dequantization saturates a 273 GB/s memory bus, while NVFP4 doesn't have to deal with that. Over a couple of days of usage, vLLM drafted 367,937 times and accepted 1,014,296 of the 1,839,636 tokens it speculated, which works out at a mean accepted length of 3.76 out of a possible 6 and an average draft acceptance of 55.1%. Per-position acceptance across the five-token window came in at 85%, 69%, 53%, 40%, and 30%, which is close enough to the figures DeepSeek published and suggests that this is a fairly optimal deployment. If you point the model at a coding task, it also runs considerably faster. In a dedicated coding test, I was able to sustain 76 tokens per second on a single stream through the best burst, with draft acceptance at 82.9% and per-position rates of 96%, 89%, 81%, 77%, and 72%, meaning that the drafter's first guess is right 95% of the time and its fifth is still right nearly three quarters of the time. It isn't even the fastest I saw in my logs, as it climbed to 106.7 tokens per second in another workload. Real agentic usage sits goes below those peaks, but it's still perfectly usable. In a session with something like Hermes that mixes tool calls, structured output, reasoning traces, and file edits, my 55% average acceptance is more representative. Draft acceptance is a function of how predictable the next token is, and agentic work alternates between highly repetitive structures like function calls and JSON and completely unpredictable reasoning tokens. It's not an exact science You need to test a lot of configs The Nvidia forums have a lot of different configurations that you can try out for all kinds of models and all kinds of clusters. while they got me most of the way there, a few omissions stopped it from starting at all. The ConnectX-7 links had no IP addresses, the suggested NCCL interface configuration made the two nodes choose different interfaces, and the download strategy had both machines independently pulling 160 GB of weights over Wi-Fi. It's a great forum, but you'll need to understand what's happening, still. Instead, I built my own pipeline that handles all of it. One machine downloads the weights using parallel workers to pull the 160 GB model weights, then it shares them to the second node over the ConnectX-7 connection in just a few minutes. There were a few other fixed I had to deploy as well, like NCCL needing a 64 GB shared-memory allocation and unlimited memlock. As well, I limited the GPUs to 2000 MHz, a pretty significant drop from the boost clock of 3000 MHz. Interestingly, this had next to no impact on token generation, but it did noticeably cut down power usage. ConnectX-7 does more than add bandwidth It's completely RDMA Setting up both devices over ConnectX-7 was pretty simple, but it's still an odd communication setup. The GB10 utilizes ConnectX-7 through two PCIe Gen5 x4 links rather than a single x8, so each physical QSFP cage maps to two OS interfaces and each one is capped to 100 Gb/s. However, you don't actually need any of that to get one model onto two machines. Before the cable arrived I connected the boxes over 10 GbE using llama.cpp's RPC backend. It pooled enough memory to load DeepSeek V4 Flash, but it split the layers between them: one Spark ran its half, then waited while the other ran the rest. You get pooled memory but not pooled compute, and a faster cable can't help an idling GPU. Tensor parallelism is what makes both boxes work at the same time, and it's the one that needs ConnectX-7. Each layer requires both nodes to combine their partial results before the next layer can start, twice per layer, so on a 43-layer model like DeepSeek V4 Flash, that's 86 synchronisation points in every forward pass. ConnectX-7 also does RDMA, which means the card writes straight into the other machine's memory without the processor being involved at all, whereas ordinary Ethernet has no such path and each of those handshakes gets copied into a staging buffer, pushed through the operating system's networking stack, and copied out again at the far end. Using the 10 GbE port on the back of each Spark, I wired the two together to find out how much of a difference it really makes. I pointed NCCL at the Ethernet port, turned RDMA off and reloaded the model. I wasn't expecting it to work, but it actually came online in just a few minutes. However, as it turns out, it is a lot slower. I scaffolded an entirely random prompt so that the drafter had nothing to predict, and tested it against both configurations. When connected via ConnectX-7, decode was a fairly healthy 17.1 tokens per second compared to the 9.2 tokens per second using 10 GbE. As well, prompt processing fell from 1,888 to about 770. What's interesting, though, is that decode fell almost as steeply as prefill, when decode barely puts anything across the cable at all, which tells us how important RDMA is. An 8 KB message takes 3.4 microseconds over RDMA and 74.6 over Ethernet, while plain TCP over the ConnectX-7 cable comes in at 14.2, meaning that the significant increase comes from the transport itself. And that happens 86 times every forward pass. Bandwidth does somewhat matter, but only for prompts, as each handshake carried as much the model can be given at once. Here, vLLM works through prompts in 8,192-token chunks, making each handshake roughly 67 MB. This means that a 120,000-token session that you might run with an agent pushes something like 90 GB across the fabric before the model has produced a single token of output. Loading a large repository into context is the entire reason to want a setup like this, as it will massively speed up your prompt processing stage and your decoding stage. DeepSeek put its prices up, but the API is still cheaper It's hard to make back $9,449 a few cents at a time DeepSeek is raising its prices, but, as I'm sure most people have guessed, that doesn't justify the cost in a setup like this. Even after the increase, off-peak rates are $0.007 per million input tokens with a cache hit, $0.22 per million input with a cache miss, and $0.66 per million output tokens, with peak rates exactly double across all three. On a conventional 80% input / 20% output coding-workload assumption, with 80% of input hitting cache, DeepSeek costs about $0.17 per million total tokens off-peak. On paper, that puts $9,449 of Sparks at roughly 55 billion API tokens, or 18 months at 100 million tokens a day, and that doesn't account for electricity. On top of that, over three days, my cluster took in 184.1 million prompt tokens against 1.38 million generated, which is 99.3% input by volume, and every turn re-sends the entire conversation back to the model. Prefix caching runs far above 80% on a coding agent as well, and mine hit 95.8%, which drags the effective rate down to about $0.02 per million. Three days of agentic work would have cost $3.79 on the API, or $1.40 a day, which will cover your $9,449 in roughly eighteen years. However, there are still real gains for a setup like this, with the primary one being privacy. Every token generated stays within your network, and you can run a coding agent, Hermes, or anything else without having to rely on a cloud provider seeing your queries and the generated responses. On top of that, you can use these devices to fine-tune or train your own models, which is what they're really good at. With all of that said, once you have the hardware, actually running the model is incredibly cheap. Idle power costs more than the inference does Electricity costs are cheaper, at least I logged the pair through a smart plug for three days of agentic use, sampling every ten seconds, and power usage was, almost surprisingly, nothing really to do with tokens in the way I expected. In fact, two thirds of the electricity went on doing nothing at all. When both machines were idle, the two pulled 78W, and under sustained inference, they average 231W and peak at 272W. The 2000 MHz clock lock is doing a lot here, as before I applied it, the same workloads averaged 259W and spiked to 324W. That means capping the clocks took about 11% off of the sustained power draw and 16% off the peak power draw while also costing nothing in performance. Over 65 hours, both machines pulled a total 6.33 kWh, which at Irish rates (€0.21/kWh from 11PM to 8AM, €0.43 during the 5PM to 7PM peak, €0.40 the rest of the time) came to €2.07: Time Cost Active 8.7 hours €0.67 Idle 56.4 hours €1.41 Total 65.1 hours €2.07 In other words, 68% of the electricity bill was the cost of having both machines available instantly. In those 65 hours of usage, vLLM served 4,716 requests, taking in 184.1 million prompt tokens with 95.8% served from the prefix cache, leaving 7.43 million to compute, and generating 1.38 million output tokens. Priced on DeepSeek's off-peak rates that exact workload costs $3.79, or about €3.26, so running the Sparks around the clock came out about 1.6 times cheaper than the API's new cheapest tier, and roughly a third of what peak pricing would have cost. However, if I turned them off between sessions, the same work would cost €0.67, which is 4.9 times cheaper than the API off-peak and nearly ten times cheaper than peak. If I built a wake-on-LAN script to manage these machines so that they were only on when needed, I'd save quite a bit of the expense. There's still a hard limit on token generation though. Given that I'm averaging roughly 50 output tokens per second, both machines can generate about 4.3 million tokens a day. Prefill isn't so much of an issue given that it computed 7.43 million uncached prompt tokens at an effective 1,366 t/s in a cumulative 90 minutes. However, decode definitely is one, and if you need substantially more than 4.3 million output tokens a day, the answer is to either use the API or get more powerful hardware. Some parts still need work It's hard to find the right setup Unfortunately there are still some problems with this setup, and some of them are quite painful to work out. vLLM, for example, has a bug with DeepSeek V4's hybrid KV topology that breaks prefix-cache persistence across sessions. Caching works within a session, but once the client's context disappears, the cached blocks effectively become unreachable. The issue has been open on GitHub since May, but it doesn't look like it's getting fixed anytime soon. Neither container image I'm using here is official, either. These are community-made builds from, ultimately, strangers, and while they work, it's hard to rely on something like that if you wanted to deploy this in a more mission-critical setup. Finally, at $9,449 for two DGX Spark machines, this isn't exactly impulse-buy territory. It's for someone burning a serious number of tokens or someone who needs privacy. Think of development done under confidentiality agreements, for example. I can't actually ascertain whether it's a good buy or not, as the uncertainty is more than just the numbers. Quantization and speculative decoding could make the next model worth running fit comfortably on one Spark, making two look excessive. Frontier models could just as easily keep growing until 256 GB isn't enough. Anyone claiming to know which way that goes is wrong, as the current AI landscape is vastly different to what it was six months ago. It'll be vastly different in the next six months, too. With this, you're buying capability that's usable today but held against the standards of a roadmap that nobody knows the future of. Local inference has gone from "it kind of works" to "it's competitive" in a short space of time, and it's hard to ever know whether there's risk of a plateau, or if things will continue growing from here. Even now, whether it competes on inference price alone depends almost entirely on how often you remember to turn the things off.