MiniCPM-V 4.7 35B-A3B EXL3 6.08bpw on ExLlamaV3, one Spark, thinking off
1× DGX Spark · ExLlamaV3 ·
Tested configuration
- Hardware
- 1x NVIDIA DGX Spark (128GB)
- Serving runtime
- ExLlamaV3
- Runtime version or build
- r0b0tlab's ExLlamaV3 fork (branch minicpm-v4.7-dspark @ 5ddae7d12448, ExLlamaV3 v1.5.0 base with sm_121 fixes and the MiniCPM-V 4.7 text architecture), built from source into a host-native venv with torch 2.14.0+cu130 and the host's CUDA 13 toolkit
- Weight quantization or precision
- EXL3 6.08bpw
- Benchmark date
- Recipe credit
- r0b0tlab MiniCPM-V 4.7 EXL3 GB10 recipeBy r0b0tlab
b8859be8b257ddb7602f68ce7b85a7f0fd03f7b4
Decode sweep
~256 input / 512 output tokens per request. Output throughput is aggregate across concurrent requests. TTFT means time to first token.
| Concurrency | Output tok/s | TTFT p95 (s) |
|---|---|---|
| 1 | 47.74 | 2.74 |
| 2 | 52.42 | 11.40 |
| 4 | 52.07 | 31.15 |
| 8 | 52.56 | 69.86 |
| 12 | 51.66 | 110.22 |
| 16 | 52.74 | 145.74 |
Prefill and context
- Configured request window
- 262,144 tokens
- Longest tested prompt
- 245,760 target input tokens
- Validated retrieval
- ≈240K input tokens · passkey passed
Input throughput uses full request wall time. The passkey measures retrieval from one long prompt.
| Workload (c1) | Input tok/s | TTFT p95 (s) |
|---|---|---|
| 4,096 input tokens | 1,450.52 | 1.61 |
| 32,768 input tokens | 2,344.87 | 13.22 |
| 131,072 input tokens | 1,567.20 | 82.15 |
| 245,760 input tokensPasskey prompt | 1,082.46 | 226.64 |
Sustained load
20-minute soak at concurrency 8 · ~1024 input / 512 output tokens per request.
- Aggregate throughput
- 51.35 output tok/s
- TTFT p95
- 72.65 s
- Completed requests
- 128
- Request errors
- 0
Serving regression checks 13/13 passed
Tool-use evaluation
Incomplete pilot · 98.6% completionSafety gate failed
Comparisons require complete results with 100% completion under the same protocol ID. This run is retained as individual evidence.
- Protocol ID
tool-eval-standard-v1-b60e2f0- Workload
- 69 mock-tool scenarios per trial · 3 trials
- Mean score
- 86.0 / 100
- Pass@3 · passed at least once
- 79.4%
- Pass³ · passed every trial
- 79.4%