Some 40% of companies now spend more than $10 million a year on AI, according to a recent report. Most of that money is not going toward training new models. It is going toward inference, the continuous work of running trained models against live data. AI inference cost now represents 85% of the enterprise AI budget, according to AnalyticsWeek’s 2026 Inference Economics report.
Inference bills scale with usage, and usage is unpredictable, which is why finance and IT teams are increasingly caught off guard by swings in AI spend from one month to the next. Inferencing costs are spiking even as token unit price declines, due to several factors including the exponentially larger token requirements to run agentic AI, the expanded ingestion of internal data for AI projects, aka the Rag Context Tax and the adoption of continuously-running AI processes versus on demand.
Eye-popping AI bills have given rise to FinOps for AI, the discipline of tracking and controlling AI-related token, inference, and GPU spend so it ties back to measurable business value, as defined by The SaaS Library. Common KPIs are in development but these three from The Saas Library are useful starting points, with the last one hardest to attain: cost per token, cost per inference and cost per outcome.
FinOps for AI borrows the accountability model that cloud FinOps built for compute and storage, but AI spend behaves differently. It is billed by the token, the GPU hour and the API call, and it is greatly impacted by the data that is ingested into models.
Organizations building FinOps for AI programs are converging on five areas of focus to rein in costs. Four are becoming familiar. The fifth, data management, is still underestimated yet it may offer the largest and most durable savings.
Sizing GPU capacity to real demand
The most direct lever is GPU utilization, a highly visible form of AI waste. Average GPU utilization across enterprise clusters sits at just 5%, according to a 2026 Cast AI report covering 23,000 clusters. IT infrastructure teams and the CIO can reduce this waste by scheduling workloads more tightly, sharing GPU pools across projects, and matching instance size to actual model demand instead of provisioning for peak load.
Matching workloads to the right infrastructure
Not every AI workload suits the same infrastructure. Training runs, batch inference and real time inference each carry different latency, throughput and cost profiles. IT teams are moving to a hybrid AI infrastructure across cloud, on premises and edge devices to place each workload where it runs most economically. This decision sits primarily with infrastructure leaders and the CIO, made in coordination with whoever owns the AI roadmap.
Optimizing inference processes
Below the infrastructure layer sits the model and runtime layer, where data engineering and AI teams do their work. Industry analysts estimate that 55% to 80% of enterprise AI GPU spend now goes to inference, not training, which is why this layer carries so much of the savings opportunity. Caching frequent queries, batching requests, routing simpler queries to smaller models that use less GPU and running models at lower precision can cut managed inference spend substantially without changing output quality in ways users would notice. This is the layer where a data engineering team, or a newly formed AI team reporting to a chief data officer, has the most influence.
Treating data management as a cost lever
Data management, especially for unstructured data that is large and constitutes 80-90% of all data, may be the more consequential lever for AI FinOps over time. Most of the data that organizations want to feed into AI systems is unstructured: files, documents, images, video and sensor output scattered across storage systems with widely-varying quality and hidden context. Sending petabytes of such dark data into an AI pipeline without first understanding it is cost prohibitive not to mention wasteful: IT is spending to transfer, store and process this data, whether or not it actually is required for the project.
Granular classification and curation of that data allows data engineering teams to separate what is genuinely useful to an AI project from low-quality, irrelevant and even harmful data. Metadata enrichment, the process of tagging files with attributes denoting content, projects and owners, sensitivity, age and relevance, turns petabytes of unstructured data into a searchable and actionable asset.
Once data is classified this way, IT can cull redundant, obsolete and trivial data (ROT) out of the way while protecting as needed sensitive data such as PII and IP. Temporary copies created for AI processing can be deleted automatically from cloud storage once a job finishes, instead of steadily accumulating as a forgotten cost. Automated data workflow technologies can continually discover, tag and enrich data, send curated data sets to AI, further enrich based on the results, and then delete copies after processing has completed. That’s an efficiency boon for IT infrastructure teams due to the sky-high cost of storage today.
Further, data teams can use unstructured data management to reduce their bills and deliver largely-untapped file data sets to their tools. This means viewing rich metadata in tables from withing their favorite lakehouse platform, so they can easily decipher what they need before importing large raw data sets that further inflate AI and analytics budgets needlessly.
With policies and close monitoring, unstructured data management for AI inferencing addresses two numbers budget owners track separately: it shrinks the storage bill, and it shrinks the AI processing bill, since smaller, better targeted datasets cost far less to run through a model and often produce more relevant results.
Building continuous visibility into spend for CFOs
None of the four levers above hold up for a CFO without a way to see them. FinOps for AI depends on continuous visibility into spend: allocating cost by project, team or business unit, flagging anomalies before a bill arrives rather than after, and reviewing inference cost on a tighter cadence than training cost, since inference spend can drift week to week while training spend tends to move in larger, more predictable steps. This is what turns AI spend from a surprise on the monthly invoice into a number the CFO can forecast and defend. Naturally, there are now specialized tools for AI spend tracking to make all this easier for finance departments.
What it means for the business
Put together, these five areas turn AI cost from an open-ended risk into a more transparent, predictable line item. The CIO gets infrastructure sized to actual demand instead of worst-case demand. The data engineering team and chief data officer get a data estate that is searchable to support AI work instead of creating a black hole. The CFO gets a spend forecast that holds up from one month to the next. And the business gets the thing all three are ultimately serving: room to fund the next AI project without cutting something else to pay for the last one.
As AI usage keeps climbing, the complexity of tracking and lowering costs is bound to grow. Yet with new AI FinOps metrics to track and analytical processes to manage both IT resources and data for optimal processing and storage, enterprise IT can reduce the burden on the budget while expanding use cases across the business.
