Context, Cost, & AI: Strategies to Optimise Agent Performance & ROI

After months of experimenting with large language models (LLMs), enterprises are moving from isolated pilots to broad deployments. However, as adoption speeds up, many tech leaders find that the biggest challenge isn't selecting the right AI model but managing the cost of optimising it.

While AI models might be getting more expensive owing to geopolitical tensions and expanding use cases, the added challenge is that the price per token continues to rise. The problem is that enterprise AI is using a lot more tokens as companies scale new applications across different departments.

In the recent episode of the Don’t Panic It’s Just Data podcast, host Kevin Petrie, VP of Research at BARC, sat down with Eudald Camprubi, Co-Founder of Nuclia (acquired by Progress) and Software Fellow at Progress and Michael Marolda, Senior Product Marketing Manager, Agentic RAG. 

They talked about how the next phase of enterprise AI will focus less on model selection and more on context engineering, retrieval strategies, and governance.

What’s the Actual Cost of Scaling Enterprise AI

Several enterprises started their AI journey with isolated proofs-of-concept. However, scaling such experiments revealed a new operational challenge.

"We're moving from enterprises experimenting with AI and doing small projects to scaling it across their business," says Marolda. "They're really focusing on what those costs will be, understanding repeatable patterns, and finding ways to save."

Unlike traditional software licensing, AI costs change with usage. Every prompt, retrieval, and generated response uses tokens, making costs harder to predict as adoption spreads across marketing, legal, customer support, and engineering teams.

Marolda says many business users are encountering AI for the first time and lack an understanding of what "normal" token consumption looks like.

"We talk to line-of-business users who haven't really touched AI before," he explained. "When they think about token costs, they have no idea what appropriate spending looks like or how to set up repeatable patterns."

For enterprise leaders, this creates a governance challenge as much as a technology one.

Bigger Context Windows Don't Always Produce Better AI

One misconception that drives unnecessary costs is the belief that larger context windows automatically enhance AI performance. According to Camprubí, many enterprises initially viewed expanding context windows as a chance to send entire knowledge bases to an LLM.

"There was a misunderstanding from customers," he explained. "They thought: let me send as much context as I can because the context window is so big."

That approach worked while token prices were still relatively low. Today, however, enterprises are realising that indiscriminately sending more information increases costs without necessarily improving accuracy.

Instead, Camprubí emphasised the need to send only the information an AI model truly requires.

"We see enterprises asking how they can ensure they send the right context to the LLM," he says. "They want to reduce the context to cut back on token consumption."

This change has introduced context engineering as a strategic discipline. 

Why Context Engineering Is Becoming Enterprise AI's Competitive Advantage

The discussion about enterprise AI often focuses on foundation models, but Progress argues the real differentiator is context engineering.

"The context is the information you send to the LLM to provide the answer," the co-founder of Nuclia explained.

Generating high-quality context requires more than just vector search. It involves intelligently combining semantic search, keyword retrieval, knowledge graphs, and user intent to surface only the most relevant information.

"If you don't send the right context to the LLM," Camprubí warned, "it will start to hallucinate, generate frustration, and create a poor experience."

For enterprise AI initiatives, improving retrieval quality brings businesses two key benefits – lower token consumption and higher answer quality. This combination directly boosts AI ROI.

Meanwhile, Marolda believes retrieval strategies will become one of the key features that separate mature AI programs from experimental ones. Rather than using the same search method for every task, enterprises should customise retrieval approaches for specific business needs.

A legal team searching for a specific contractual clause has different retrieval requirements than a sales executive preparing for a customer meeting.

"Tailored retrieval strategies for each specific use case will become increasingly important," Marolda told Petrie.

Instead of sticking to one retrieval method, enterprises are now combining semantic understanding with keyword search and structured enterprise knowledge to produce more accurate responses while minimising unnecessary context. The result is improved performance, reduced costs, and more reliable AI outputs.

What Does the Future of Enterprise AI Rely on?

As enterprises progress beyond experimentation, success will rely less on picking the next game-changing model and more on using existing models wisely. That means providing precise context instead of excessive context.

Enterprises should select the right model instead of just the latest model. That means they should treat observability, retrieval strategies, and governance as foundational capabilities rather than second priorities.

Camprubí said that enterprise leaders should be wary of industry hype. "Don't trust everything you read on LinkedIn.” In enterprise AI, measurable results will determine which enterprises effectively scale intelligent systems.

Listen to the full episode of Don't Panic! It's Just Data to hear Michael Marolda and Eudald Camprubí discuss Agentic RAG, token optimisation, context engineering, and the future of enterprise AI at scale.

Takeaways

• Token costs are climbing, making cost management critical for enterprises.
• Context is essential for effective AI implementation and user satisfaction.
• Enterprises must focus on providing the right context to LLMs to avoid hallucinations.
• Modularity in AI platforms allows for flexibility and adaptability in solutions.
• Quality metrics are vital for evaluating AI outputs and ensuring reliability.
• The latest AI models are not always the best choice for every use case.
• Understanding user intent is crucial for effective data retrieval.
• FinOps teams are increasingly involved in managing AI costs and strategies.
• Enterprises should consider RAG as a service to reduce maintenance burdens.
• Collaboration among stakeholders is essential for successful AI implementation.

Chapters

• 00:00 Introduction to AI and Data Context
• 03:26 Understanding Token Costs in AI
• 09:03 The Importance of Context in AI
• 14:37 Exploring the Context Layer and Retrieval Strategies
• 22:31 Model Selection and Cost Management in AI
• 29:11 Key Takeaways for AI Leaders

For more enterprise AI, Agentic RAG, data governance, and enterprise knowledge layer insights, follow Progress Software across its official channels:

• Website: Progress Software
• YouTube: @ProgressSW
• LinkedIn: Progress Software
• X: @ProgressSW