Qwen3.8-Flash-Next on Eight GPUs: What Actually Helped

Aug 27, 2026

We ran Qwen's Qwen4 preview across eight RTX PRO 6000 GPUs. Speculative decoding helped at low load, but smarter routing cut repeat-request startup from 2.5 seconds to 287 ms.

Qwen3.8-Flash-Next landed with the kind of spec sheet that makes inference engineers cancel their afternoon. It has a 125B-parameter main model with 6B active per token, a 51B table for two- and three-token patterns, and a 4B multi-token prediction module that helps it generate several tokens at once. It also accepts images and holds up to 262K tokens of context. The whole model is a preview of the architecture Qwen is exploring for Qwen4.

The official Qwen release and vLLM's day-zero recipe gave us a strong starting point. We wanted to answer a more practical question: what does it take to serve the official FP8 checkpoint well on one eight-GPU RTX PRO 6000 server?

We ended up running two copies of the model. Each copy was spread across four GPUs and served by vLLM. In front of them sat ramjet, our load balancer, which tries to send repeat work back to the engine that already has the shared prompt in memory. Then we changed one thing at a time.

Conceptual, not tensor-by-tensor. Architecture labels come from the official FP8 model card and technical report.

The model combines several ways of remembering context. Most layers maintain a compact running summary using Gated DeltaNet. Every fourth layer can look farther back using Qwen Sparse Attention. Near the start of the model, a large lookup table recognizes common two- and three-token patterns. The model also contains 512 specialist subnetworks, but selects only 10 of them, plus one shared expert, for each token. This is how a 125B main model can activate roughly 6B parameters at a time.

A separate multi-token prediction module proposes the next few tokens. The main model checks those proposals together instead of generating every token in a separate step. This technique, called speculative decoding, can make an individual response much faster when the proposals are good.

The server

The node has eight RTX PRO 6000 Blackwell Server Edition GPUs: 768 GB of aggregate GDDR7 ECC capacity and roughly 12.8 TB/s of aggregate local-memory bandwidth. That bandwidth is the sum of each card's local memory—not a claim about GPU-to-GPU links. Each card is configurable up to 600 W, giving the box a 4.8 kW GPU nameplate ceiling. Thermals are a serving constraint, not a footnote.

The official FP8 checkpoint is 172.78 GiB. Spreading one copy across four GPUs gives it 384 GB of physical memory for the weights, working memory and the key-value cache that remembers tokens already processed. Two copies consume all eight GPUs, but each keeps its own process and cache. The vLLM recipe supports this four-GPU arrangement, often called tensor parallelism or TP4, where memory allows. The two-copy RTX deployment described here is our configuration, not an upstream reference design. Hardware specifications come from NVIDIA's product page and 8-GPU reference architecture.

This is the same practical idea behind the Helix Sovereign Server deployment guide. Keep model weights, prompts, tool calls and agent traces on infrastructure you control. Then expose enough information to see whether the system is healthy and where requests are going.

What worked

First, the basic layout worked. Both model copies passed our tests for tool calling, reasoning, images and long prompts, including one containing 251K tokens. In a load test capped at 64 simultaneous requests, all 128 test requests completed. Together, the two engines generated 3,340.5 tokens per second, with the work split evenly between them.

The useful tuning results were less obvious:

The routing result needs a picture. Imagine one engine is answering a long request and already holds that request's prompt in memory. The other engine is idle. When related work arrives, sending it to the first engine lets it reuse the prompt; sending it to the idle engine means reading the prompt again.

Our old router made the wrong choice. It continued to count the busy engine as if it were still reading the long prompt, even after it had started generating the answer. That made the idle engine appear cheaper, so the related request lost its warm cache.

Ramjet now lowers that load estimate as soon as the model starts answering. The related request stays with the cached prompt, and its wait for the first output token falls from 2.5 seconds to 287 milliseconds. The long request retained 99.1% of its original throughput. In a separate test with 32 simultaneous requests, total throughput also rose by 2.7%.

The largest gain did not come from a more aggressive kernel flag. It came from teaching the router when the expensive part of a request had actually ended.

The next comparison: GLM-5.3 Flash

GLM-5.3 Flash arrived with a similar efficiency-first pitch, but it activates more of the model for each generated token. Z.ai reports 320B total parameters and 18B active per token. Qwen reports a 125B main model, 51B of pattern-matching embeddings and a 4B prediction module, or about 180B across those components, with 6B main parameters active per token.

These are overlapping benchmark names from the vendors' own launch evaluations, not a controlled head-to-head. DeepSWE alone used different sampling settings and context limits; the NL2Repo guardrails and context limits also differ. The comparison is useful directionally, but the deltas are not our measurements. Sources: Z.ai's GLM-5.3 Flash launch post, official GLM model card, and official Qwen model card.

Qwen also published scores for several other agent and software-engineering tests:

Source: Qwen's model card. These benchmarks use different harnesses and task definitions, so the bars describe a profile rather than a single comparable scale. DeepSWE reports the better of two harnesses, SWE-bench Pro uses Qwen's refined task set, and CoWorkBench is Qwen's in-house benchmark.

Our own checks asked narrower operational questions. Did tool calls arrive in the expected format? Did the reasoning and final answer land in the right response fields? Did image input work? Could the model retrieve facts from a very long prompt? Did the engine's token counts agree with the client? Passing meant “safe enough for more serving experiments,” not “we independently proved the model intelligent.”

What did not work

Letting vLLM process more prompt tokens in each batch sounded promising. Raising max-num-batched-tokens from 8,192 to 16,384 barely changed the typical cold-start wait. The average wait grew by 29%, however, and the slowest 5% of requests waited 88% longer. We rolled it back.

More cache hits were not automatically better either. We made ramjet more willing to send work to a busy engine if that engine already held the prompt. Cache reuse rose from 35.9% to 47.9%, but repeat requests started 15.1% slower and the long request they shared the engine with generated 7.3% fewer tokens per second. We kept the original balance.

The vLLM recipe also offered to move cached data into ordinary system RAM. Its generated setting asked for roughly 220 GiB for each GPU worker, or 880 GiB for one four-GPU model copy, on a machine with 125 GiB of RAM. Moving the large token-pattern table to system RAM had the same basic problem: the two-engine stack did not leave enough headroom to do it safely. Both stayed on the GPUs.

The bugs were useful

The first boot, limited to 85% of GPU memory, failed safely because nothing remained for the cache of previously processed tokens. At 90%, vLLM's automatic sizing offered 38.32 GiB for that cache on one boot and only 4.48 GiB on an otherwise equivalent warm boot. We stopped trusting the automatic value and pinned the first allocation that had booted and passed our tests.

The OpenAI-compatible cached_tokens field also stayed at zero even while vLLM's own metrics counted reused prompt tokens. Our tests now read the engine metrics instead of trusting that response field for this preview model and runtime.

Our rollback code caught us once too. It restored the Compose file but forgot the exact ramjet image we had been testing, so an older, incompatible load balancer briefly started. The model engines never changed and the balancer refused traffic rather than guessing. Every deployment and rollback command now names the exact image. We also collected the usual test-harness scars: one doubled /v1, one wrong base path, and an image test that stopped halfway through the model's reasoning because its output limit was too low.

Where we left it

The final configuration uses the official FP8 weights and keeps the token cache on the GPUs. It predicts up to three tokens ahead, reuses sparse-attention work between those prediction steps, and limits each scheduler batch to 8,192 prompt tokens. Ramjet sends related work back to the engine with the reusable prompt, while reducing that engine's estimated load once it starts answering.

We built another routing mode that tries to predict the load each engine will carry, but left it disabled because we did not finish testing it.

When the air entering the chassis rose above our safety limit, we stopped testing instead of forcing another benchmark run. The model is paused while the server moves to its next experiment. That is part of the result too: a good serving setup includes a clear point at which you stop tuning.

The full Compose deployment, experiment journal, and router source are public.

Measured on an 8× RTX PRO 6000 Blackwell Server Edition server using Qwen/Qwen3.8-Flash-Next-FP8. We ran two vLLM model copies, each spread across four GPUs and kept close to its CPU and memory socket, behind ramjet. These are serving measurements from this machine and workload, not general model-quality claims.