thelocalbuild
Field notes
500K tokens of context, on four boxes you can fit on a desk
A measured on-premise deployment of GLM-5.2 across four DGX Spark nodes, adaptive speculative depth, full CUDA graph coverage, and every number published with its denominator.
Most "long context" claims are about an API ceiling somebody typed into a config file. This one is different: an exact 500,000-token cold prompt was pushed through a four-node DGX Spark cluster running GLM-5.2, it completed, and the whole run was instrumented. The interesting part is not the headline figure. It is that the project publishes what it did not establish alongside what it did, which is exactly the standard on-premise infrastructure should be held to.
Here is what the deployment does, why it is technically hard, and what it tells you about running frontier-class open models on hardware you own.
The deployment at a glance
| Component | Configuration |
|---|---|
| Hardware | 4 × DGX Spark nodes, 1 × GB10 each, ARM64 (sm_121a) |
| Interconnect | RoCE / NCCL over RDMA |
| Model | GLM-5.2, QuantTrio/GLM-5.2-Int4-Int8Mix |
| KV cache | NVFP4, DeepSeek-MLA layout |
| Parallelism | TP4 × DCP2 × PP1 × DP1 |
| API context | 520,000 tokens |
| Physical KV pool | 525,887 tokens (8.41 GB) |
| Max concurrency | 3 requests (max_num_seqs = 3) |
Four Sparks. No datacentre, no rack of H100s, no cloud. That is the point, this is a class of hardware a mid-sized team can actually put in a cupboard, and it is running a frontier-class open-weight model at half a million tokens of context.
Why long context is the hard part
Scaling context is not a software flag. Every token you accept has to be stored as key/value state for the entire life of the request, and that state lives in GPU memory next to the weights. On this deployment the measured slope is ≈ 15.98 KiB per logical token per rank. Multiply that out and a half-million-token conversation is an 8.4 GB standing allocation that never gets smaller while the request is alive.
Two compression choices make it fit. The weights use Int4/Int8 mixed quantisation, and the KV cache itself is stored in NVFP4 using a DeepSeek-MLA layout. Note the asymmetry that the project is careful to state: NVFP4 applies to the cache only, not the weights. If you have read our guide on running open-source LLMs on-premise, this is the same VRAM arithmetic, just at the edge of what the hardware can hold.
Adaptive multi-token prediction: the depth ladder
Speculative decoding makes generation faster by having the model draft several tokens ahead and then verify them in one pass. Accepted drafts are free speed. Rejected drafts are wasted compute. Most implementations pick a fixed draft depth, say K=5, and eat the waste whenever acceptance drops.
This build makes the depth adaptive, but on a discrete ladder of K=2, K=4 and K=5. It starts at the floor and climbs on evidence:
| At depth | Observation | Action |
|---|---|---|
| K2 | Head acceptance ≥ 0.85 over 32 steps | Probe K4 |
| K2 | Head acceptance < 0.85 | Hold at the floor |
| K4 | (p2 + p3) accepted per batch ≥ 0.70 | Probe K5 |
| K4 | Tail gain 0.35 – 0.70 | Hold |
| K4 | Tail gain < 0.35 | Retreat to K2 |
| K5 | p4 accepted per batch ≥ 0.15 | Hold |
| K5 | p4 < 0.15, tail ≥ 0.35 | Fall to K4 |
| K5 | p4 < 0.15, tail < 0.35 | Retreat to K2 |
The design detail worth stealing: transitions are governed by unconditional marginal gain, what the fourth and fifth draft positions specifically earn, not by the overall acceptance ratio. A whole-prefix ratio flatters low depths, because dropping K shrinks the denominator. Measuring the marginal position asks the only question that matters: is this extra draft slot paying for itself?
The controller never descends below K=2, and K=3 is unreachable: it is not a rung on the ladder.
The CUDA graph problem nobody mentions
Here is where adaptive depth usually falls apart. CUDA graphs capture a fixed sequence of GPU operations so the driver stops re-planning work on every step; on small-batch decode they are a large share of your throughput. But a captured graph is tied to an exact tensor shape. Change the draft depth and you change the token count per step, and an uncaptured shape silently drops to a piecewise or eager path that is materially slower.
So an adaptive controller can easily hand back in graph downgrades everything it saved in avoided drafts, and you would never see it in a benchmark that only reports tokens per second.
The fix is unglamorous and correct: enumerate every reachable shape and capture all of them. With three depths and up to three concurrent sequences, that is nine full decode graphs.
| Depth | Query length | Token counts captured |
|---|---|---|
| K = 2 | 3 | 3, 6, 9 |
| K = 4 | 5 | 5, 10, 15 |
| K = 5 | 6 | 6, 12, 18 |
Coverage is asserted fail-closed at load time: if any reachable depth lacks a captured graph, the runtime refuses to start rather than quietly degrading. Live logs across all four ranks show cudagraph_mode resolved to FULL_AND_PIECEWISE, zero piecewise downgrades, zero uncaptured-shape warnings, zero eager fallbacks.
It is not free. Capturing nine shapes instead of three costs 2.14–2.25 GiB of graph pool per rank, against 1.01 GiB for the fixed-K=5 predecessor, and pushes capture time from 3 s to 24 s. Hold that number, it comes back later.
What the 500K run actually measured
One isolated request, concurrency 1, against the live 520K profile. The prompt was counted through the server's own /tokenize endpoint at exactly 500,000 tokens, carried a unique nonce so the prefix cache could not replay it (measured cache-hit delta: 0), and the completion was capped at 128 tokens with ignore_eos so the run could not stop early and look faster.
| Metric | Value |
|---|---|
| Prefill | 500,000 KV tokens in 972.400 s = 514.192 tok/s |
| Time to first token | 973.196 s |
| Decode | 33.430 tok/s |
| End to end | 977.245 s |
| Peak KV occupancy | 95.107 % |
| Preemptions / restarts / Xid events | 0 / 0 / 0 |
| Draft acceptance | 82.86 %, mean 4.143 accepted positions per batch |
| Graph mode | FULL across all nine shapes |
Read the shape of that, not just the numbers. Time to first token exceeded sixteen minutes. This is a demonstration of cold-prompt capacity, not an interactive latency target. If your workload is a chat assistant, this run tells you nothing about your experience. If your workload is "ingest an entire contract set, codebase or case file in one pass, overnight, without any of it leaving your network", that is precisely what it proves.
Note also that during this window the controller sat at K5 for all 32 steps. At 82.86 % acceptance the tail positions were earning their keep, and the policy holds K5 exactly when they do. The controller not moving is the correct outcome, not a broken one.
The controller does move under real traffic
Evidence for adaptation comes from an earlier live window on a 550K profile, across 32 telemetry windows:
| Quantity | Value |
|---|---|
| Scheduler steps at K2 / K4 / K5 | 177 / 64 / 783 |
| Draft batches | 2,162 |
| Drafted → accepted tokens | 9,202 → 6,210 (0.675 per drafted) |
| Measured average selected K | 4.256 |
| Fixed-K5 comparator, measured | 4.965 – 5.000 |
| Draft positions avoided | 14.9 % |
Per-position unconditional gain fell off exactly as the policy assumes, 0.895, 0.747, 0.483, 0.411, 0.335 from p0 to p4, which is why the fifth slot is the one worth arguing about. And critically, the fixed-K5 baseline was measured at 4.965–5.000 rather than assumed to be 5.0, which is what makes it a legitimate comparator.
The memory floor, and the OOM that set the envelope
The most instructive section of the whole project is the one about failure. An earlier build ran the same stack at a 550,000-token profile. It served for 3 minutes 20 seconds and then the Linux kernel OOM-killed the head node.
The cause was measured, not guessed. The head node carries the API server, the engine core process and the Ray head, and had 647 MB of host memory free before cutover. The nine-shape graph coverage added +0.92 GiB per rank over the fixed-K build. It fit at load, it fit through graph capture, and then had nothing left for peak activation when concurrent requests arrived. The three workers had 3.1–3.5 GB and would have absorbed it fine.
During the successful 500K run, minimum available memory on the head node was 484.6 MiB, and it hit that floor 37 seconds into prefill at just 4.11 % KV occupancy. The binding constraint is the activation and kernel-workspace transient, not the final cache fill. SwapTotal is 0 on all four nodes, so a spike is an instant kill, not a slowdown.
Projecting the measured slope forward, +5K tokens costs ~78 MiB per rank and leaves a ~406 MiB floor; +20K leaves ~172 MiB. Against a sane 512 MiB OS reserve there is zero evidence-backed room to grow. So the profile was cut to 520K. That is what engineering discipline looks like: the spec follows the measurement, not the marketing.
What is deliberately not claimed
This is the section that should make you trust the rest, and it is rarer than it should be:
- Adaptive is faster than fixed K5 in tokens/second, not established. No valid matched sample exists; the A/B harness is built but the arm errored on its first case. Avoiding 14.9 % of draft positions is a measurement of speculative work avoided, not of speed.
- A general decode-speed figure, not established. 33.430 tok/s is one request, concurrency 1, at 500K context, on this exact profile.
- Depth-transition cost, not measured. Switching depth switches CUDA graphs; oscillation overhead is unknown.
- Total KV savings, no counter exists on this build, so no number is offered. The preallocated pool is measurably unchanged.
- Any result on non-GB10 hardware, not attempted.
There is also an honest architectural limitation: depth is chosen per batch, not per request. All concurrent requests share one K, so a low-acceptance stream can drag everyone else's depth down. On a three-concurrency box that is tolerable. On a busy multi-tenant server it would not be.
What this means if you are buying on-prem AI
Three takeaways generalise well past this specific stack.
Long context is a memory-engineering problem, not a model problem. Whoever sells you an on-premise server should be able to state the KV slope per token, the concurrency it supports at full context, and how much host memory is left at peak. If they cannot, they have not run it at the limit.
Ask which numbers are measured and which are projected. A vendor quoting tokens per second without concurrency, context length and denominators is quoting a vibe. Note how much of the analysis above only becomes possible because the failure, the OOM, was published too.
Frontier-class context now fits in small hardware. Four Sparks is not a datacentre. The gap between "we need a cloud API for this" and "we can run this ourselves" keeps closing, and for anyone weighing that decision, our breakdown of on-premise vs cloud AI covers the cost and compliance side.
Frequently asked questions
- Can you really run a 500,000-token context window on-premise?
- Yes. Four DGX Spark nodes running GLM-5.2 with Int4/Int8 weights and an NVFP4 KV cache served an exact 500,000-token cold prompt at a 520,000-token API ceiling, hitting 95.107 % peak KV occupancy with zero preemptions. Time to first token was over 16 minutes, so treat it as batch capacity, not interactive latency.
- What is adaptive multi-token prediction?
- Instead of always drafting a fixed number of tokens ahead, the scheduler moves between K=2, K=4 and K=5 based on whether the extra draft positions are actually being accepted. Here it produced a measured average depth of 4.256 and avoided 14.9 % of draft positions versus a verified fixed-K=5 baseline.
- Doesn't varying the draft depth break CUDA graphs?
- It normally does, each depth produces different tensor shapes, and uncaptured shapes fall back to a slower path that can erase the savings. This build enumerates all nine reachable shapes (3/6/9, 5/10/15, 6/12/18) and captures each as a full decode graph, verified fail-closed at load with zero downgrades across all four ranks.
- Does this run on other hardware?
- Not as published. It is ARM64 and GB10-specific (
sm_121a), and no other topology has been validated. The techniques (adaptive draft depth, exhaustive graph capture, publishing denominators) transfer; the numbers do not.
Source
All figures above are from the public GLM-5.2 R9 Adaptive MTP repository and its docs/BENCHMARKS.md, which carries the full methodology, receipts and caveats. Model weights are licensed separately.