GLM-5.3 Flash EXL3 4 bpw: TensorFold v0.6.0 with the DFlash2 drafter, two Sparks, 1M (recipe 3fd60dfc)
2× DGX Spark · TensorFold ·
Tested configuration
- Hardware
- 2x NVIDIA DGX Spark (128GB each)
- Serving runtime
- TensorFold
- Runtime version or build
- TensorFold v0.6.0 (commit c4646171) plus the recipe's patches, on nvcr.io/nvidia/pytorch:26.07-py3
- Weight quantization or precision
- EXL3 4 bpw
- Benchmark date
- Recipe credit
- MiaAI-Lab TensorFold recipeBy MiaAI-Lab
3fd60dfc836fab0f1ed0fd4f0afaf2a1d62b2aab
Decode sweep
~256 input / 512 output tokens per request. Output throughput is aggregate across concurrent requests. TTFT means time to first token.
| Concurrency | Output tok/s | TTFT p95 (s) |
|---|---|---|
| 1 | 54.53 | 0.44 |
| 2 | 74.56 | 0.66 |
| 4 | 93.20 | 0.91 |
| 8 | 91.86 | 24.45 |
| 12 | 85.14 | 52.92 |
| 16 | 90.76 | 74.50 |
Prefill and context
- Configured request window
- 1,048,576 tokens
- Longest tested prompt
- 131,072 target input tokens
- Validated retrieval
- UnavailablePasskey not run
Input throughput uses full request wall time. The passkey measures retrieval from one long prompt.
| Workload (c1) | Input tok/s | TTFT p95 (s) |
|---|---|---|
| 4,096 input tokens | 882.50 | 4.34 |
| 32,768 input tokens | 1,744.96 | 17.36 |
| 131,072 input tokens | 1,777.92 | 74.18 |
Sustained load
20-minute soak at concurrency 8 · ~1024 input / 512 output tokens per request.
- Aggregate throughput
- 80.18 output tok/s
- TTFT p95
- 29.98 s
- Completed requests
- 194
- Request errors
- 0
Serving regression checks 12/12 passed
Tool-use evaluation
Complete · 100% completionSafety gate failed
Compare only against complete results with 100% completion under the same protocol ID.
- Protocol ID
tool-eval-standard-v1-b60e2f0- Workload
- 69 mock-tool scenarios per trial · 3 trials
- Mean score
- 94.0 / 100
- Pass@3 · passed at least once
- 92.8%
- Pass³ · passed every trial
- 92.8%