A Busy GPU Is Not a Productive GPU: New Scorecard Rewrites AI Inference Metrics
NAND Research and WEKA's Inference Economics Scorecard argues GPU utilization hides wasted context rebuilding. An 8-hour HGX B300 test showed 3.7x more tokens per GPU-hour with shared NVMe cache.

Why it matters
- WEKA and NAND Research propose the Inference Economics Scorecard, measuring useful throughput against power, rack space, and cost instead of GPU utilization.
- An eight-hour benchmark on an eight-GPU NVIDIA HGX B300 node running DeepSeek-V4-Pro showed 3.7x more output tokens per GPU-hour with a shared NVMe cache.
- Context recomputation consumed 60% of prefill cycles in the HBM-only baseline, versus 6.8% with the shared cache.
- 97% of successful requests hit first token within five seconds with the shared cache, versus 44% for the baseline.
- The test replayed over 5,000 real agent traces with 512 active sessions and 32 concurrent requests; WEKA sponsored the research.
An eight-hour benchmark on NVIDIA HGX B300 hardware delivered 3.7 times more output tokens per GPU-hour with a shared NVMe cache than with HBM-only serving. That number anchors The Inference Economics Scorecard, new research from NAND Research and WEKA that attacks one of the industry's favorite dashboard numbers: GPU utilization.
The core argument is blunt. "A busy GPU is not necessarily a productive GPU." Utilization cannot distinguish productive output from expensive context rebuilding. An inference server can burn substantial compute reconstructing earlier conversation state while its utilization metrics still look healthy.
The stakes are real. AI agents now maintain long conversations, call tools, and return repeatedly to earlier context. As that behavior becomes the norm, the way operators measure infrastructure economics has to change with it. The scorecard is a proposed evaluation framework, not an established industry standard — and WEKA, whose storage technology benefits from this framing, sponsored the research.
Why GPU utilization hides expensive repeated work
Inference runs a trained model to produce an answer. Before generating new tokens, the model processes input and conversation history in a phase called prefill. During that work it builds a key-value cache, or KV cache: intermediate attention data that lets the model reuse earlier computation as the conversation continues.
The fastest home for that cache is the GPU's high-bandwidth memory, HBM. But HBM must also hold model weights and other working data. Long sessions and competing users can force the server to evict cached context to make room.
When a user or agent returns, the missing cache may need to be rebuilt by reprocessing old tokens. The model has not permanently lost the conversation — the infrastructure has discarded a reusable representation and must pay to reconstruct it. That creates the measurement trap at the heart of the paper: reprocessing yesterday's context keeps the GPU just as active as generating a useful new answer. Utilization remains a valuable diagnostic signal, but by itself it cannot tell an operator how much productive work customers actually received.
What does the scorecard actually measure?
The framework demands continuous tests lasting at least eight hours, long enough to expose behavior short demonstrations miss. Comparisons must hold the model, precision, workload, concurrency, cache conditions, quality threshold, and service-level objectives constant. The measurement boundary covers the serving stack, with cooling and facility overhead disclosed.
Only successful output meeting quality and service targets counts toward useful throughput. Failed requests, retries, timeouts, and discarded output must stay visible rather than vanishing from the calculation. The scorecard's metrics include:
- Tokens per watt — how efficiently power converts into useful model output
- Tokens per rack — output delivered within a physical footprint
- Cost per inference — the expense of serving a successful request across the system
- GPU goodput — productive output within service targets, rather than raw chip activity
- Time to first token and end-to-end latency — how long users wait for a response to begin and finish
- Memory efficiency — how effectively the system uses scarce memory capacity
- Operational density — how deployment and operating constraints affect usable capacity
- Agent concurrency — how well the system supports multiple active agent workloads
- Context retention — how effectively reusable session context survives as workloads grow
The throughput-versus-goodput distinction matters most here. A server that produces more tokens but misses a customer's response deadline can be less useful than a slower-looking system that consistently meets it. Likewise, a larger cache only pays off economically if its benefit exceeds the added storage, networking, and operating expense.
How shared storage extends the KV cache
WEKA's technical account describes NeuralMesh with Augmented Memory Grid, which keeps reusable KV data on shared NVMe storage beyond the GPU's own memory. NVMe is a high-performance interface commonly used with flash drives. In this design the cache becomes a resource the serving infrastructure retrieves across the network, instead of something repeatedly reconstructed on a GPU.
RDMA, or remote direct memory access, and NVIDIA GPUDirect Storage move data along a direct storage-to-GPU path with less CPU involvement. The economics reduce to a trade between moving existing data and recomputing it. A sufficiently fast storage path can make retrieval cheaper than rebuilding long histories — but the advantage depends on cache reuse, context length, concurrency, and the speed of the entire data path. A short request whose working data already fits in HBM presents a different problem entirely.
What did the eight-hour benchmark find?
The joint Nebius and WEKA test ran one eight-GPU HGX B300 serving node on DeepSeek-V4-Pro. The replay workload drew on more than 5,000 real agent traces, held a pool of 512 active sessions, and used a scheduler permitting 32 concurrent requests. (Active sessions and simultaneous requests are different measures.)
Across eight hours, WEKA reported:
- 3.7x more output tokens per GPU-hour with the shared cache than with HBM-only serving
- 97% of successfully completed requests reached a first token within five seconds, versus 44% for the baseline
- Recomputing previously encountered context consumed 60% of prefill cycles in the baseline, versus 6.8% with the shared cache
The caveats are significant. The additional cache used NVMe drives spread across an eight-node cluster, even though the reported serving result came from one GPU node. The GPU-normalized gain therefore does not establish an equivalent reduction in total cost or energy — those calculations must include the supporting resources. The baseline also matters: the technical write-up says a suitable DRAM-offload implementation for this model was unavailable when testing took place. The results compare the tested configuration with HBM-only serving, not proof of superiority over every alternative cache architecture.
Do open traces make the comparison more useful?
WEKA thinks so. Its open-source kvlab project captures and replays the structure of agent sessions — including timing and cache-reuse patterns — without distributing the original conversation or source-code content. Its loader is documented in NVIDIA's AIPerf tooling, and the announcement also identifies integration with SemiAnalysis's AgentX benchmark.
Preserving pauses, growing histories, and related agent activity exposes cache pressure that a stream of unrelated synthetic prompts would overlook. The limits are worth noting: such replays measure serving behavior, not whether an agent completed a business task correctly.
For enterprises, the scorecard's central contribution is a clearer definition of success — useful responses, at the required quality and speed, within the resources available. As agents consume more context and run longer, a busy GPU becomes merely a starting observation. What matters commercially is the work that reaches the customer.
Source: unite.ai
More from Elena Vasquez
Show full bio
Market editor covering technology and product news in artificial intelligence and machine learning at AI In Context.
708 articles
Related articles
- Agentic AI Workloads Could Quadruple Data Center CPU Core Demand
- AWS Adds KV Cache-Aware Routing to SageMaker AI Endpoints
- Google Releases DiffusionGemma, a 26B Model That Generates Text Four Times Faster
- OpenAI cuts GPT-6 agent costs with smarter prompt caching
- OpenAI Taps Cerebras for 750MW of Compute to Speed Up ChatGPT