The Hidden Cost of AI: Why Enterprises Are Struggling to Connect Spending to Business Value
Enterprises are rapidly becoming adept at tracking their AI expenditures, yet they remain largely unable to articulate the tangible return on that investment. As the initial excitement surrounding generative AI shifts toward operational maturity, organizational leaders are finding that visibility into raw billing data does not equate to an understanding of business outcomes. This disconnect has created a significant "economics of AI" crisis, marked by a sharp pivot from the unbridled experimentation seen in early 2024 to a more cautious, budget-constrained environment today.
During a recent SD Times Live! Supercast centered on the economics of AI, J.R. Storment, executive director of the Tokenomics Foundation, described a profound market whiplash. The first half of the year was defined by a phenomenon Storment termed "tokenmaxxing," where organizations prioritized speed and deployment over cost-efficiency. By spring, the mood had soured as the lack of a clear causal link between AI consumption and revenue generation forced companies to impose strict spending caps.
Data from the Tokenomics Foundation’s recent survey underscores this trend. While respondents reported a high level of confidence in their ability to track AI costs—knowing exactly how much they were spending on providers like OpenAI, Anthropic, or Google—the confidence plummeted when asked to justify those costs. Only 9% of respondents could confidently map their AI spending to verifiable business outcomes. As Storment noted during the discussion, the response to inquiries regarding value realization was "crickets."
The Vibe-Based Management Problem
The difficulty in measuring AI ROI is not limited to mid-market firms; it is pervasive across the industry, including within the infrastructure organizations that support open-source development. Shawn Alpay, director of data engineering at the Linux Foundation, admitted that even his team struggles to move beyond "vibes" when evaluating the effectiveness of their AI implementations.
For many engineering teams, the primary metric for AI success has been throughput—how many pull requests are generated or how much code is suggested. However, Alpay pointed out that higher throughput does not necessarily equate to higher quality. In fact, a deluge of AI-generated code can often create a bottleneck, making it harder for human engineers to review, debug, and merge changes. "Teams compare notes on new models, but it’s all vibes," Alpay explained. There is a systemic failure to quantify whether a change in model architecture reduces token consumption while maintaining or improving code quality.
This issue is exacerbated by the lack of standardized attribution. Many enterprises pull AI services from four to six different sources, each with its own idiosyncratic billing structure. As Storment emphasized, "a token is not a token." Without a unified way to track usage, companies cannot allocate costs to specific business units or departments, leading to a "black box" of operational expense.
The Technical Path to Financial Visibility
To address these attribution challenges, the industry is looking toward standardization. Alpay is currently working to steward FOCUS (FinOps Open Cost and Usage Specification), an initiative designed to unify billing data across disparate cloud providers. However, standard billing data is insufficient; it lacks the granular context of what specific application workload triggered a model request.
To close this gap, the FOCUS project is collaborating with OpenTelemetry (OTel)—a project under the Cloud Native Computing Foundation—during its 1.6 development cycle. The goal is to build a telemetry schema that allows organizations to bridge the gap between OTel trace data and FOCUS cost data. This would, for the first time, allow engineers to see the "cost per request" or "cost per trace," effectively tying every cent spent to a specific action within the software stack.
Furthermore, industry experts caution that common cost-cutting measures can be counterproductive. For instance, the popular strategy of routing requests to smaller, cheaper models often destroys the benefit of cached tokens. If a system is optimized to use a specific model’s cache, switching to a cheaper alternative can actually increase costs because the system is forced to re-process and re-encode data from scratch.
The Shift to Agentic Workloads
The complexity of AI cost management has reached a breaking point due to the rise of autonomous agents. Unlike static chatbots, agents operate in loops, calling external tools and making iterative decisions about how much data to process. Jerod Johnson, a technology evangelist at CData Software, noted that modern AI costs are now driven by agent behavior rather than human user input.
CData Labs conducted a rigorous benchmark study to understand these dynamics, running 27 different models through a standard enterprise workflow: reading, ranking, and inserting data into a review queue. The results were stark. Without proper data context, the models struggled significantly, with only one out of 400 runs succeeding. When business context was provided, however, 12 of the models achieved a 100% success rate.
The economic implications are clear: the cost of a "failed" run is infinite because the task remains incomplete. In the CData study, a run without context cost $350 and yielded only one correct answer. With the right data and architecture, a similar task cost only $93 and produced 318 correct answers, bringing the cost per correct answer down to approximately 29 cents.
Context, Turns, and Model Architecture
Johnson frames the economics of AI using a simple but powerful formula: Cost = Context × Turns × Model. The "times" sign is the most critical variable. When an organization fails to provide sufficient context, the agent must perform more "turns" (iterations) to reach a conclusion. Each turn consumes more tokens and increases the latency of the operation.
According to CData’s research, providing the right context reduced the number of tool calls from 40 to 8 and minimized the number of runs hitting the 40-turn cap from 37% down to 2%. This suggests that the "AI bill" is not primarily a model-choice problem, but a data-architecture problem.
"If you get your data and your architecture right, the model choice just becomes a budgeting decision," Johnson argued. Organizations that invest in preparing high-quality, structured data to feed their agents will inevitably see lower costs and higher performance than those that rely on "prompt engineering" to fix poorly architected data workflows.
Implications for Engineering Leadership
As companies move into the next phase of AI adoption, leadership must shift its focus from procurement to governance. Simply tracking total spend is no longer sufficient. Leaders must now ask their teams to answer three fundamental questions:
- What was the specific business question or task this AI workload was meant to resolve?
- How many iterations (turns) were required to reach a verified outcome?
- Which data path was utilized, and was that the most efficient way to achieve the result?
The era of "vibes-based" AI investment is drawing to a close. As the market matures, the competitive advantage will go to organizations that treat AI as a rigorous engineering discipline rather than a novelty. By integrating telemetry with financial data and optimizing the context provided to agents, companies can move away from the "whiplash" of unpredictable costs and toward a sustainable model of AI-driven value creation.
Ultimately, the goal is to reach a point where every AI dollar is accounted for. As Shawn Alpay advised, while the spirit of innovation and building is vital, the "trust but verify" mantra must be applied to the budget with the same intensity applied to the code. Until organizations can move from measuring tokens to measuring "cost per correct answer," the true economic potential of generative AI will remain elusive.
