Profiling DGX Spark: where a token spends its time

This started with netrunner’s post about MLX on a Mac. According to the author, large expert-weight buffers were tripping a batching threshold even when an operation needed only a small slice. Changing how those slices were presented avoided unnecessary submissions.

That raised a question for my own machine: does the Spark keep doing useful GPU work during generation, or does it repeatedly stop and wait for the CPU?

The follow-up reported a 25% improvement across several changes, without isolating that one fix’s contribution. I treated it as an investigation to learn from. It was not a prediction for a different model, runtime, and GPU.

I started with a model already on disk: Qwopus3.6-27B-v2, Q4_K_M, running through the existing CUDA build of llama.cpp. Its reported tensor size is about 16.8 GB, which left comfortable memory headroom. This is a dense model; it does not exercise the expert routing in the MoE model from the original post.

Hardware

NVIDIA DGX Spark · GB10 · ARM64

Runtime

llama.cpp · b22ff4b

Execution

GPU layers offloaded · Flash Attention on · 8 CPU threads

Timing

64 generated tokens · depth zero · 3 repeats per configuration

Capture

Nsight Systems 2025.3.2 · 32 generated tokens + warmup

Analysis

Final 20 CUDA graph replays · startup excluded

llama-bench uses synthetic tokens here. This measures the inference path, not answer quality, a real chat session, or long-context performance. The unprofiled runs establish throughput; the instrumented run helps explain the work.

On a Mac, the analogous starting point is a Metal trace in Xcode or Instruments. On the Spark, Nsight Systems shows the CPU/GPU timeline; Nsight Compute inspects individual CUDA kernels. I started with Systems.

Here is an actual step from the middle of the selected window. One CPU graph launch schedules a graph containing 1,813 GPU kernels. The kernel count alone would be misleading: these are not 1,813 separate CPU submissions.

The long quantized matrix-vector operations are where most of the recorded kernel time goes. Across all 20 selected replays, Q4_K and Q6_K variants of mul_mat_vec_q account for 93.77% of summed kernel duration. That gives the next investigation a specific target.

The selected window lasts 1,576.945 ms. Taking the union of kernel and copy intervals gives 1,559.402 ms of GPU activity: 98.89% of the window. Only 17.543 ms is uncovered, and the largest individual gap is about 0.881 ms.

There is a trap in the arithmetic. Adding every kernel’s duration produces 1,609.744 ms—more than the window itself—because kernels overlap across streams. The 98.89% figure merges overlapping intervals first. The 93.77% figure answers a different question: which kernel family dominates the sum of recorded kernel durations?

Another trap is the CPU API summary. cudaStreamSynchronize occupies a lot of CPU time, but a CPU waiting for the GPU is not automatically a synchronization bug. In this trace the GPU is usually executing work while that wait happens.

None of this proves that memory bandwidth or compute units are saturated. A running kernel can still be inefficient. The trace rules down idle gaps as the main opportunity in this workload; it does not explain the internal limits of the expensive kernels.

CUDA graphs let a runtime replay a sequence of GPU operations with less CPU submission work. To see how much that helped here, I ran the normal configuration, disabled graphs for one process with GGML_CUDA_DISABLE_GRAPHS=1, then repeated the normal configuration.

The two normal runs agree closely. Comparing their mean throughput with graphs off gives a difference of about 0.36%. This small A/B/A exercise provides little reason to chase a large launch-batching win on this particular path.

The traced 32-token run measured 12.55 tok/s. I kept it out of the table: instrumentation adds overhead, and its sequence length differs. It is not a clean measurement of profiler overhead.

With a CUDA-enabled llama.cpp build, compatible Nsight Systems installation, a local model that fits, and an idle GPU, the capture command is small. Set MODEL to your GGUF file and BENCH to your llama-bench binary.

nsys profile --trace=cuda,nvtx --sample=none \
  --cuda-graph-trace=node -o decode-trace \
  "$BENCH" -m "$MODEL" -p 0 -n 32 -r 1 \
  -ngl 99 -fa 1 -t 8 -o json

Open the resulting .nsys-rep in Nsight Systems. Separate initialization and graph construction from the repeated decode steps before drawing conclusions. Node-level capture shows individual kernels, but it can add significant overhead.

Download the plotted measurements and kernel intervals (JSON). This is a compact export of the measurements used here, not the full profiler recording.

The first trace narrowed the question. Most of the time is inside quantized matrix-vector kernels, and there is little empty space between GPU operations. I have not made the model faster.

The next step is a targeted Nsight Compute capture of those kernels: DRAM throughput, cache behavior, compute throughput, occupancy, and stalls. The current driver restricts those counters to administrators, so they were not collected in this experiment. Without them, memory bandwidth versus dequantization or other computation remains unresolved.

A useful optimization would need a concrete explanation, numerical correctness checks, and an improvement in unprofiled inference. The larger MoE model also needs its own trace in its actual serving runtime. This dense, short-context result cannot stand in for that.

For now, I have a repeatable way to look inside a decode step—and a much better reason to investigate one kernel family instead of changing flags at random.

• netrunner: the original MLX batching investigation and the performance follow-up
• NVIDIA: Nsight tools on DGX Spark
• NVIDIA: diagnosing CUDA graph performance
• NVIDIA: Nsight Compute profiling guide
• cuda.fast on DGX Spark: real speedup, narrow claim
• Qwen3.8-Flash-Next: three runtimes, one box