Spark Bench

MiniCPM-V 4.7 35B-A3B EXL3 6.08bpw on ExLlamaV3, one Spark, thinking off

1× DGX Spark · ExLlamaV3 ·

Tested configuration

Hardware
1x NVIDIA DGX Spark (128GB)
Serving runtime
ExLlamaV3
Runtime version or build
r0b0tlab's ExLlamaV3 fork (branch minicpm-v4.7-dspark @ 5ddae7d12448, ExLlamaV3 v1.5.0 base with sm_121 fixes and the MiniCPM-V 4.7 text architecture), built from source into a host-native venv with torch 2.14.0+cu130 and the host's CUDA 13 toolkit
Weight quantization or precision
EXL3 6.08bpw
Benchmark date
Recipe credit
r0b0tlab MiniCPM-V 4.7 EXL3 GB10 recipeBy r0b0tlabb8859be8b257ddb7602f68ce7b85a7f0fd03f7b4

Decode sweep

~256 input / 512 output tokens per request. Output throughput is aggregate across concurrent requests. TTFT means time to first token.

ConcurrencyOutput tok/sTTFT p95 (s)
147.742.74
252.4211.40
452.0731.15
852.5669.86
1251.66110.22
1652.74145.74

Prefill and context

Configured request window
262,144 tokens
Longest tested prompt
245,760 target input tokens
Validated retrieval
≈240K input tokens · passkey passed

Input throughput uses full request wall time. The passkey measures retrieval from one long prompt.

Workload (c1)Input tok/sTTFT p95 (s)
4,096 input tokens1,450.521.61
32,768 input tokens2,344.8713.22
131,072 input tokens1,567.2082.15
245,760 input tokensPasskey prompt1,082.46226.64

Sustained load

20-minute soak at concurrency 8 · ~1024 input / 512 output tokens per request.

Aggregate throughput
51.35 output tok/s
TTFT p95
72.65 s
Completed requests
128
Request errors
0

Serving regression checks 13/13 passed

Tool-use evaluation

Incomplete pilot · 98.6% completionSafety gate failed

Comparisons require complete results with 100% completion under the same protocol ID. This run is retained as individual evidence.

Protocol ID
tool-eval-standard-v1-b60e2f0
Workload
69 mock-tool scenarios per trial · 3 trials
Mean score
86.0 / 100
Pass@3 · passed at least once
79.4%
Pass³ · passed every trial
79.4%

Read the tool-eval report and scenario findings