Kubernetes clusters are running at an average CPU utilisation of 8%. Not 8% at night, not 8% on weekends. 8% across the full year, measured across tens of thousands of organisations on AWS, GCP, and Azure. That is from Cast AI’s 2026 State of Kubernetes Optimisation Report, which is based on measured production-cluster data rather than survey responses. It suggests that a large share of reserved compute in production is not being translated into useful work while the cloud bill compounds.
That is the baseline before AI workloads enter the picture. The same report found average memory utilisation at 20%, down from 23% the prior year, and average GPU utilisation at 5% across the enterprise Kubernetes clusters analysed.
Cloud-based infrastructure made provisioning radically easier. In doing so, it removed most of the friction that historically forced efficiency. Organisations no longer need to plan capacity months in advance, justify hardware purchases, or live with the consequences of a bad forecast. Teams can simply provision more. The result is a culture of overprovisioning baked into how engineering teams work: set resource requests high at sprint planning, rarely revisit them, and treat unused headroom as an acceptable cost of reliability.
For most of the past decade, that trade-off was manageable. The waste was real but diffuse, spread across CPU and memory allocations that were hard to measure and easy to ignore. FinOps emerged as a discipline to surface and manage that waste. It helped. But it largely treated the symptom rather than the cause and infrastructure that is structurally difficult to optimise at the speed at which it changes. The report also found that CPU overprovisioning rose from 40% to 69% year on year, while memory overprovisioning stood at 79% – a sign that the gap between provisioned capacity and what workloads request is widening, not closing.
AI workloads change the economics
With AI infrastructure, the same structural problem gets a much higher cost multiplier.
GPU compute is expensive by definition. As demand for accelerated compute rises, underused capacity is no longer just an operational inefficiency. It is a margin problem. Across the enterprise Kubernetes clusters analysed in the report, average GPU utilisation was 5%, measured as the share of provisioned GPU compute cycles producing useful output over a 24-hour period. The report excludes AI lab clusters, where GPU use is typically more intensive. In practical terms, provisioned GPU capacity is running far ahead of useful GPU compute output.
The demand profile makes it worse. With AI inference, a dynamic replica count is not enough: by default, 1 GPU = 1 model. When capacity is full, a new GPU cannot always be added quickly, because capacity may not be available. Static provisioning, which was already the wrong model for Kubernetes, is a particularly poor fit for AI infrastructure but remains common because there are few easy options. This is why GPU sharing is gaining traction. For small models, or large GPUs, MIG and other GPU-sharing techniques are becoming the equivalent of VPA and HPA for GPUs.
Then there is token optimisation. The variance between model tiers can be an order of magnitude on per-token pricing. Organisations that default every request to a top-tier model when cheaper tiers would meet the quality bar for many requests are not making a technical trade-off; they are structurally overspending in a way that is difficult to detect without request-level cost attribution and SLO scoring. The standard FinOps tooling built for cloud computing has not kept pace with the costs of LLM APIs.
Why the optimisation gap keeps growing
The gap between what organisations are spending and what they should be spending is not closing. It is widening. There are structural reasons for that.
First, the feedback loop is too slow. By the time a cost anomaly surfaces in a dashboard or a quarterly review, the waste has already occurred, often for weeks. Cloud costs are a real-time problem being handled as a batch problem.
Second, the people who understand workload behaviour do not own the cloud budget, and the people who own the budget do not have the technical context to know which pods are overprovisioned, which models are consuming expensive capacity, or which workloads could move to more efficient placement. The optimisation work that falls between those two groups gets nobody’s time. Better dashboards do not close that gap. More frequent reviews do not close it either. The only thing that closes it is moving the optimisation loop from periodic human review to continuous automated adjustment.
Third, the complexity is genuinely hard. At scale, organisations are managing hundreds of nodes, multiple GPU instance types, and dozens of workloads with different demand profiles. Manual optimisation is not a process; it is a fiction. The surface area is too large and changes too fast for any team to cover with spreadsheets and monthly reviews. The report’s Spot analysis illustrates the point, fewer than 2% of GPUs ran on Spot Instances through most of 2025, while availability and pricing varied sharply by region and instance type.
Efficiency as a financial imperative
The first phase of cloud adoption was about speed, organisations deploy faster, scale faster, move faster. Infrastructure efficiency was a secondary concern. Most organisations still operate on that mental model.
The second phase, already underway for organisations with significant AI infrastructure, focuses on efficiency. Not because efficiency has become more virtuous, but because the cost of inefficiency has crossed a threshold where it materially affects margins, capital allocation, and competitive position.
The practical response is using automation to right-size resources, optimise instance selection, route workloads to appropriate compute tiers, and match GPU allocation to actual demand. In the report, organisations that applied automated rightsizing reduced their provisioned CPU footprint by approximately 50% on average; that is a footprint reduction, not a claim that total cloud spend falls by the same amount.
AI workloads are generating some of the most significant cloud cost pressure the industry has seen, and AI-driven automation is one of the few practical ways to address it at the scale modern infrastructure operates. Manual optimisation cannot keep pace with the rate of change. The loop has to close continuously, not quarterly.
What this means for finance and engineering leadership
The shift underway is not primarily technical. It is organisational.
Engineering teams have historically been rewarded for shipping fast and maintaining reliability. Cost efficiency was someone else’s problem or, at worst, a constraint applied after the fact. That incentive structure made sense when compute was cheap relative to the value of engineering velocity.
The organisations that will have an advantage in the next phase of AI adoption are not necessarily the ones with the largest infrastructure budgets. They are the ones that close the loop between infrastructure decisions and financial outcomes fastest.
That requires changing what engineering teams measure, what they are held accountable for, and how quickly the feedback between spend and behaviour reaches the people who can act on it. The tooling and automation have to catch up to the speed at which AI infrastructure costs are moving.
The next phase of AI adoption will not be defined by how much companies spend on infrastructure. It will be defined by how efficiently they use it.
Token economics
The world is increasingly running on tokens. Every prompt, every completion, and every agent loop consumes them. And as AI adoption accelerates, token consumption is becoming a more visible line item in cloud budgets, often without the cost-visibility infrastructure that organisations spent the last decade building for CPU and memory.
The problem starts at the model tier. Frontier models from the major labs are priced at a level that assumes the use case justifies the quality premium. Many do not. A large fraction of enterprise inference workloads – summarisation, classification, extraction, internal tooling and routing logic – are tasks where a well-tuned open-source model may deliver sufficient results at a fraction of the per-token cost. The difference is not always marginal. Organisations defaulting every request to a frontier model because that is what the demo used are not always making a considered architectural decision. They may be leaving money on the table at scale, invisibly, because nothing is tracking quality-adjusted cost per request.
Rate-limiting compounds it. When an organisation depends on a third-party API for inference, the provider controls the throughput ceiling. That ceiling does not adjust automatically to a traffic spike, product launch, or end-of-quarter batch job. The natural response from finance and engineering leadership has been to ration tokens by setting quotas, gating access, and slow-roll features. That response can feel responsible, but it risks taxing productivity rather than solving the underlying cost problem.
The more durable answer is to optimise tokens rather than simply ration them, and to run enough efficient inference capacity that rationing becomes less necessary.
Optimised tokens can come from open-source models deployed on controlled infrastructure. The model quality gap between frontier APIs and open-source alternatives has narrowed for many narrow production tasks. When inference runs on well-utilised GPU capacity, the unit economics can look very different from API-based pricing. The organisation is not paying per call; it is paying for compute capacity, and the value of that capacity depends on how efficiently it is used.
GPU fleet management is becoming a core cost discipline
That brings the discussion to GPU fleet management, where the economics become particularly important. A GPU waiting between inference requests is unproductive capacity. The utilisation profiles for AI inference follow demand curves that can be predictable at the fleet level, even when they are unpredictable at the request level, for example, business hours in one region, quiet hours in another, and an overnight batch in a third. An organisation with users across Asia, Europe, and the US is running three overlapping demand curves on the same underlying hardware. APAC peaks while the US sleeps. Europe ramps up as APAC winds down. The US takes over as Europe goes quiet. When treated as isolated capacity pools, each region appears underutilised for much of the day. Treated as a single fleet with autonomous workload distribution, the same GPUs can run at materially higher utilisation across all three. This is what autonomous GPU sharing does across Kubernetes clusters, and what kimchi.dev Enterprise implements at the inference layer. GPU time-sharing and cross-fleet routing that matches actual demand to available capacity in real time, without manual intervention.
Latency is not the only metric that matters
One objection comes up consistently in these conversations is latency. The argument goes that inference workloads are latency-sensitive and that shared, distributed GPU fleets cannot meet the response-time requirements of production applications. That concern is valid in some use cases, but it is often framed too narrowly. The metric that drives user experience and throughput economics is not only time-to-first-token. It is also the total tokens generated per second across the fleet. Infrastructure that prioritises throughput while still meeting appropriate latency targets can serve more users, process more workloads, and cost less per output token. Teams optimising for fastest possible first-token response on a lightly loaded dedicated GPU may be solving for the wrong constraint if much of the compute remains underused.
Token FinOps is the missing the framework
Token FinOps is the chapter that has not been written yet. There are no established frameworks for tracking quality-adjusted cost per request, no standard benchmarks for open-source versus frontier-model substitution rates, and no mature tooling for continuous token-routing optimisation analogous to what exists for CPU and memory. That gap is closing, but slowly. The organisations that build these capabilities now, combining open-source model deployment, controlled inference infrastructure, and autonomous GPU fleet management, are positioning themselves ahead of a cost curve that will only steepen as AI workload volumes grow. The rest risk continuing to ration tokens while wondering why AI productivity gains are not materialising at the expected pace.
