GPU Utilisation vs Productivity: Are Your GPUs Actually Doing Useful Work?

Your organisation has bought GPUs to run your AI workloads. The monitoring dashboard indicates they are busy, usage is high and most of the available capacity has been allocated. Everything seems to be working as it should. But there is a catch. T

raining jobs are taking longer than they should, inference requests are slowing down, and the team is already discussing buying more GPUs. What is really going on? GPU utilisation informs you how busy your hardware is, but doesn’t necessarily tell you how much productive work that hardware is doing. 

A GPU can be assigned to a job that is not ready to run, or waste its time processing workloads that interfere with each other. For infrastructure teams, the question is whether poor performance is a sign that they need more compute capacity, or if the capacity they have could be operating better.

What Does GPU Utilisation Actually Measure?

GPU utilisation is the measure of how busy a graphics processing unit (GPU) is throughout a time interval. It's one of the most used metrics for monitoring AI infrastructure, but to comprehend what it tells you, you need to know the difference between allocation, activity and real performance. 

GPU allocation refers to compute resources that have been allocated or reserved for a workload. If a training job is allotted four GPUs, those resources may be unavailable to other jobs even if the training project has not been started. Utilisation measures a separate thing. Graphics engine activity is the percentage of time during a monitoring interval that the graphics or compute engines of the GPU were busy. 

This is from the Data Center GPU Manager documentation from NVIDIA. Other measurements explain what is happening throughout that activity. Streaming multiprocessor (SM) activity describes how often the processing units of the GPU are busy with tasks. But those units can be waiting for data from memory, rather than doing calculations. 

SM occupancy is the number of groups of execution threads resident on those processing units divided by the maximum number of groups of execution threads that can be supported on those processing units. Higher occupancy doesn’t always equal better performance. Another interesting measure is GPU memory use. 

But it only tells you how much memory is utilised. Some AI apps pre-allocate memory, so a high reading doesn’t necessarily suggest the app is actively consuming it. The distinction is particularly crucial for inference purposes. Google Cloud's GPU autoscaling guideline says that GPU utilisation is a metric of activity, not how much work is done during that activity. 

As such, it cannot be relied upon to tell teams whether an inference service is meeting its throughput and latency criteria.

Why Busy GPUs Don't Always Mean Productive Work

Useful output does not always correspond to GPU activity for a number of reasons. Some issues exist before a task even begins, and others only arise when numerous jobs are running.

Allocated GPUs can be waiting for work

Imagine a scheduler gives a training job GPUs, but the work is still waiting for its data, its software environment or any other resource it needs. The GPUs have been allocated, however training cannot start. This problem is described by researchers in Ineffective Occupancy in Heterogeneous GPU Clusters, a study published in October 2026. 

They looked at certain examples from an Alibaba GPU cluster trace where a job was placed on a machine with enough GPU capacity but was not ready to run. The study found that 52.70% of the investigated execution records never reached the ready state, even when placement was finished and there was enough GPU capacity. 

This is a finding from a single research data set, not an estimate of GPU waste across the industry. The researchers developed readiness-aware scheduling in which the likelihood of a job being ready is taken into consideration before determining when to admit it. The right question for IT teams is not just how much capacity has been allocated. 

The period between allocation and execution may expose resources that are idle without progress.

Sharing GPUs can introduce performance interference

Sharing GPUs across workloads allows businesses to utilise capacity that would otherwise be wasted. But executing additional jobs on the same hardware also introduces potential for interference. Different workloads could compete for memory bandwidth, cache capacity and other shared resources. 

That means the GPU can be kept busier while individual jobs take longer to complete. This problem was examined for deep-learning inference in the October 2026 PairSight research paper. Its researchers found a way to forecast how workloads affect each other's latency when sharing GPU resources. 

Their solution obtained 4.3% mean latency-prediction error on six models assessed, while a naive prediction method yielded 19.0% error. Those numbers are about how accurate the predictions are, not about how fast the inference is. 

However, the research shows the necessity of considering the performance of the involved workloads in decisions about sharing GPUs.

Active hardware can still be a bottleneck

Sometimes the workload is running, but the GPU is not making progress as fast as one would expect. The processing unit may be waiting for data to come from memory. The CPU may be taking too long to prepare for the next operation, or data transfers may be slowing down execution. Software can also play a role. 

Meta, in research published in October 2026 on training across several GPU platforms, explained how tweaks to software libraries, compute processes and model architecture boosted performance on AMD accelerators. The researchers report up to 1.5x higher end-to-end training throughput over their direct-port baseline. 

That was the outcome for the setups they evaluated, but it’s an example of why hardware specs aren’t the only determinants of the performance a workload may accomplish.

Which GPU Performance Metrics Actually Matter?

There is no one GPU productivity indicator that applies to every AI application. The correct measurements to use depend on what you want the workload to accomplish. A training cluster must execute jobs efficiently. An inference platform should respond to requests fast and reliably. They both have GPUs but distinct priorities when operating.

Measuring GPU productivity for AI training

Training workloads are iterative computations that make incremental updates to an AI model. The key question for teams doing these activities is how much training progress the infrastructure achieves over time. Useful measurements are:

• Queue wait time: How long a job has to wait before it can start.
• Training throughput: The number of training steps or samples the system can handle per second.
• Job completion time: Time taken to complete the whole training task.
• Interruptions and breakdowns: How frequently work is interrupted or repeated.
• Productive GPU-hours: The amount of allocated GPU time that actually moves your training forward.

These measurements are especially relevant when jobs run over numerous GPUs. If one portion of the system fails or is delayed, progress in another may be interrupted. Meta described this difficulty in the engineering description of the Prometheus supercluster, published in October 2026. 

Meta said its Muse Wonder models reached over 85% goodput in pre-training runs and over 90% in reinforcement learning runs. Goodput is the proportion of available execution time that contributes to useful training progress. These results are representative of Meta’s workloads and infrastructure. 

More crucially, they explain why the organisation tracks effective training progress alongside hardware reliability and interruptions.

Measuring GPU productivity for AI inference

Inference is different in that the AI model is already trained. It processes incoming requests and returns responses. Imagine you have an AI assistant running for an enterprise. Perhaps a GPU is processing tonnes of queries and users are still waiting too long for answers. Useful GPU inference performance indicators include:

• Throughput: Number of requests or output tokens processed per second.
• Time to first token: The time a user must wait before the model begins to reply.
• Inter-token latency: Time between generated tokens.
• Tail latency: The time it takes for the service to respond to its slowest queries.
• Error rate: The frequency of failed requests.
• Energy per token: The amount of energy GPUs consume to generate each output token.

Energy efficiency is emerging as another metric of productive computing capacity. The October 2026 study The Token-Energy Law of LLM Decode explored the effect of memory-related work and execution time on the energy consumed per token generated. The researchers built a model using measurements from three language models running on NVIDIA Hopper and Blackwell GPUs. 

Its predictions deviated from the measured mean energy per token across the investigated scenarios by at most 3.73%. The study measured energy at the GPU board, not total data-centre use. Still, it is a great example of why businesses need to tie resource usage to actual production.

How Can Teams Tell Whether They Need More GPUs?

It is tempting to think that the answer to underperforming AI workloads is to buy more GPUs. But more hardware won't fix problems that come from scheduling, resource contention or inefficient software. Before teams can scale capacity, they need to know where work is getting stuck and if current resources are being used effectively.

Check whether allocated capacity is doing useful work

Begin with the comparison of resources allocated to workloads versus the work being done by those resources. A cluster dashboard can show that most GPUs are allocated, but that doesn’t tell you if the jobs you allocated them to have begun or are making progress. Teams should verify:

• How many allotted GPUs have active workloads?
• How long are jobs waiting to be scheduled and then executed?
• Are jobs holding resources while waiting for other requirements to be fulfilled?
• What workloads are responsible for the reported GPU activity?

It is easier to answer these questions if hardware measurements can be linked to a particular job. For example, Meta’s GPU Cluster Monitoring project integrates GPU telemetry with data from the Slurm workload scheduler. This enables the administrator to correlate hardware activity with individual jobs and users. 

Without such information, teams can see a busy GPU but not whether the workload using it is making progress as planned.

Identify what's limiting performance

Once teams know which workloads are running, they can look at the GPU activity and compare it with the outcomes those workloads are producing. If utilisation is high but throughput is low, the next thing is to find out what is taking time on the GPU. You can use profiling tools to find memory stalls, wasteful compute activities, and other performance limitations. 

If a GPU is shared by various workloads, teams should also assess performance when running jointly versus separately. This can show if resource contention is limiting the value of consolidation. Scheduling decisions deserve the same attention. A tweak that decreases idle time may look good on a utilisation monitor, but the ultimate test is whether it decreases job completion times or improves inference performance. 

The trick is to know what the limiting issue is before you change the infrastructure. Otherwise, teams risk optimising a statistic and not improving the workload.

Compare improvements against workload outcomes

Once a potential problem is found, teams need to assess whether the proposed solution actually improves performance. This implies getting a baseline before adjustments and then comparing performance under similar workload conditions. For training, the comparison might be on finished jobs, training throughput and how much GPU time is needed. 

For inference, it may centre around successful queries, response latency and output tokens/second. Cost should be part of the evaluation too. The cost per useful unit of work will provide you with a clearer story than cost per allocated GPU-hour. That might be the cost of a completed training run, or a successful inference request, depending on the workload. 

A scheduling tweak that improves utilisation but makes requests slower may not be an improvement. Similarly, if existing GPUs are already doing well and demand is still beyond their ability to cope, it can make sense to add capacity. The decision should be led by the evidence, not a goal utilisation percentage.

What Should a GPU Productivity Dashboard Show?

To be effective, a GPU productivity dashboard must link hardware activity to the workload causing it, and to the results produced by those tasks. That doesn’t imply every team needs to track every measure out there. The operator should have sufficient information to recognise a problem and determine its source.

The metrics should be looked at in conjunction. For example, high utilisation with decreasing inference throughput could indicate resource congestion or some other execution impediment. Low utilisation with large job queues may be a symptom of a scheduling or resource allocation problem, not a lack of raw computing capacity. 

Teams also need to consider the measured time span. Quick bursts of demand, quiet spells and continuous workloads might yield quite different readings. The most beneficial dashboard is one that helps operators move from identifying a performance problem to understanding what they need to examine.

Final Thoughts: Measure Useful Work Before Adding More GPUs

GPU usage is a good measurement but gives only part of the story. A dashboard might indicate that the hardware is active, but doesn’t tell you whether the training jobs are making good progress or whether or not inference services are delivering expected performance to users. 

For businesses that are investing in AI infrastructure, knowing that difference can help them avoid wasteful spending and make their capacity planning more dependable. As workloads get more complicated, the issue will not only be finding enough computing power. It'll also be knowing whether the resources already in hand are being used effectively. 

And that’s a question to bear in mind as you track the latest in AI infrastructure and GPU performance via EM360Tech. After all, the next big increase in computing capacity may not come from adding more hardware, but from squeezing more valuable work out of what you already have.