60% faster GLM-5.3-Flash prefill on four GPUs: the all-reduce that wins on two loses on four
Nov 22, 2026
Our four-GPU GLM-5.3-Flash replica read prompts no faster than our two-GPU ones. NCCL was routing every all-reduce through host memory. The PCIe IPC fix we use on two GPUs is slower on four, and NCCL's own peer-to-peer path took prompt reading from about 6,200 to 10,100 tokens per second.
We serve an uncensored GLM-5.3-Flash checkpoint for security work, split across four RTX PRO 6000 GPUs. Until this morning it read prompts at about 6,200 tokens per second, the same speed as our stock GLM-5.3-Flash copies that run on two GPUs each. Doubling the GPUs bought nothing.
Three environment variables later it reads about 10,100 tokens per second, 60% faster, with the same memory footprint, and it still passes our agent protocol tests. The fix that made our two-GPU replicas faster last month turned out to be the slower option at four GPUs.
How we picked this replica
We worked from the inference optimisation skill, a written method for agents tuning model serving. Its first step is to read what has already been measured and work out the largest gap before changing anything.
The two-GPU stock replicas had already been profiled close to the card's limits in October. The four-GPU uncensored replica never got the same treatment, and its traffic made the gap obvious:
- It served 4,511 requests in 3.5 days, with the longest prompts on the server: 167,000 tokens on average, against 87,000 on a stock replica.
- Measured as time to first token per uncached prompt token, it was the slowest of the three replicas: about 2,100 tokens per second against 3,400.
- Its engine processes held dozens of
/dev/shm/nccl-*mappings. NCCL was moving data through host memory.
Where the all-reduce time went
After every layer, the four GPUs sum their partial results. That step is the all-reduce, and while a prompt is being read it runs on 6,144 tokens at a time, about 50 MB per call.
These GPUs have no NVLink. They reach each other only through the PCIe host bridge, so NCCL stages the data through a buffer in host memory by default. SGLang also turns NCCL's cuMem allocator off, which leaves that buffer on whichever CPU socket had free memory when it was first touched. The previous post covers that placement problem in detail.
We benchmarked every available path on the same four GPUs in a disposable container, with the production engine image:
On two GPUs, our chunked PCIe IPC all-reduce beat NCCL's peer-to-peer mode, and we shipped it. On four GPUs the order reverses. NCCL peer-to-peer takes 2.39 ms, chunked IPC takes 4.4 ms, and the host-memory path we were running took 20.1 ms in the benchmark container.
The gap holds at smaller sizes too. At 24 rows, the size of a speculative-decoding verify step at four concurrent requests, NCCL peer-to-peer took 29 µs against 53 µs for untuned IPC and 146 µs for the host-memory path. IPC only wins at one row.
Why four GPUs flip the result
The IPC kernel is a one-shot all-reduce: every GPU reads the whole buffer from every other GPU and adds them up itself. NCCL's peer-to-peer mode, for a large message on one machine, normally runs a ring: each GPU sends a slice to its neighbour, the partial sums travel around, and the finished slices travel around again.
With two GPUs, both algorithms move the same 50 MB per GPU, and the one-shot finishes in a single step with no hand-offs, so it wins. With four, the one-shot reads three peers' buffers, 150 MB per GPU, while the ring moves 75 MB. Twice the data predicts about twice the time, and we measured 1.8 times.
That arithmetic is the textbook cost of each algorithm. We did not measure PCIe link traffic, so treat it as the explanation that fits the numbers, not a traced one.
The change
The engine's Compose file got these lines:
environment:
NCCL_CUMEM_ENABLE: "1" # SGLang forces 0; buffers then land on a random socket
NCCL_P2P_LEVEL: SYS # allow peer-to-peer across the PCIe host bridge
SGLANG_PCIE_IPC_MAX_NUMEL: "16384" # IPC keeps only calls of 4 rows or fewerThe IPC path stays on for the tiniest calls, the draft steps of speculative decoding (one row per request, so at most four), where it can still win. Everything else goes to NCCL.
On the two-GPU replicas, NCCL peer-to-peer needed about 1 GB of extra GPU memory and ran out at --mem-fraction-static=0.99. This replica runs at 0.93 and had room. Its key-value cache, the GPU memory that holds prompt state, kept room for 499,968 tokens.
The restart took 21.6 minutes, almost all of it loading weights. This is the only copy of the uncensored model, so it was offline for that window. We restarted at 05:13 UTC on a Sunday, after checking it had no running requests.
Results
Every prompt below was salted so the engine had nothing cached, which we confirmed from the cached-token count in each response.
A 37,000-token prompt now starts answering in 3.7 seconds instead of 6.1. A 149,000-token prompt takes 14.7 seconds. The four-GPU replica finally reads prompts faster than a two-GPU one, which manages about 6,400 tokens per second on a different checkpoint.
Generation moved less:
| Generating | Before | After |
|---|---|---|
| 4 requests at once, total tokens/s | 388 | 415 (+7%) |
| 1 request, tokens/s | 198.5 (one run) | 195.5 (median of four) |
| Short prompt, time to first token, 1 / 4 requests | 85 / 191 ms | 77 / 110 ms |
The single-request number may be slightly worse: time per generation step went from 14.8 to 15.6 ms. The baseline is a single 256-token request, so that is within run-to-run noise, but it has a plausible cause. Six-row verify calls now go to NCCL, whose kernels compete with the model for GPU cores. If a proper A/B confirms the loss, raising the IPC limit to six rows (SGLANG_PCIE_IPC_MAX_NUMEL=24576) sends those calls back to IPC and keeps everything larger on NCCL.
On live traffic, 96 to 98% of this replica's prompt tokens come from the prefix cache. The 60% applies to the rest: the first turn of each conversation, and turns where the cache was evicted. With 167,000-token average prompts, those are the turns where users wait longest.
The agent protocol tests passed on the new configuration: 5 of 5 tool-calling cases, 7 of 7 multi-turn sessions, and 4 of 4 long-context recall ladders, including needles at 256,000 and 512,000 tokens. NCCL adds numbers in a different order than the host-memory path, so outputs are not bit-identical. The benchmark's maximum error matched the old path.
What we did not test
- No interleaved control. Each configuration change costs a 21-minute cold start of the only replica of this model, so the baseline is one pass before the restart. The prefill gain is far outside run-to-run noise. The single-request generation number is not.
- The two-GPU replicas are unchanged. Chunked IPC still beats NCCL peer-to-peer at two ranks, 1.46 ms against 1.84 ms.
- Eight GPUs, or four split across two sockets. All four of our GPUs sit on one socket. The crossover point will move with the topology.
Checking your own four-GPU engine
If you run SGLang with four or more GPUs that have no NVLink, two checks take a minute.
First, see whether NCCL is using host memory. Against a running engine:
grep -c 'nccl-' /proc/$(pgrep -f 'sglang.*scheduler' | head -1)/numa_mapsA non-zero count means the all-reduce goes through /dev/shm. The N0= and N1= fields on those lines show which socket holds the pages.
Second, benchmark the paths on your own topology before picking one. Our script, ar_tp4.py, runs under torchrun with four ranks. It compares NCCL, with whatever NCCL_* environment you pass, against the FlashInfer PCIe IPC one-shot at 1 to 6,144 rows. Our two results point in opposite directions on the same server, so neither one transfers to your hardware without a measurement.
Methodology
- Hardware: four RTX PRO 6000 Blackwell Server Edition GPUs (96 GB, SM120), connected only by PCIe through one host bridge, on one socket of a two-socket Intel Xeon 6505P server. Full clocks at 600 W.
- Model and engine:
orcarouter/GLM-5.3-Flash-Uncensored-FP8at revision0480f690, SGLang v0.4.3 (the ormandj SM120 build), tensor parallel 4, EAGLE speculative decoding with an adaptive draft length. - Microbenchmark: BF16, hidden size 4,096, median of three, in a disposable container on the production image with the engine idle.
- Engine measurements: direct to the engine, not through the load balancer, under our thermal guard, intake air at 30 °C. Fresh salts per run, zero cached tokens confirmed from response usage.
- Artifacts: the Compose file, the microbenchmark and the experiment journal are in helixml/ramjet#332. The deployment recipe is
deploy/glm53_unc_fp8.
The load balancer in front of these replicas is Ramjet, our open-source router for prefix-cache-aware inference. The two-GPU recipe and its chunked all-reduce patch are in deploy/glm53_flash_sm120, and the October profiling work is in 9% faster GLM-5.3-Flash prefill with GPU-to-GPU IPC.