With enterprises now rushing to integrate AI agents into their operations and security, the most imperative focus now becomes the AI model itself. However, Eric Tschetter, Chief Architect at Imply, believes the real challenge is within the data infrastructure that supports these systems.

In the recent episode of the Tech Transformed podcast, Kevin Petrie, BARC Vice President of Research, sat down with Tschetter to talk about how AI is actually increasing the current needs around scale, performance, and data access.

“Agents are always running queries. They’re always doing stuff,” Tschetter stated. 

Unlike human analysts, AI systems work continuously, producing much higher query volumes and putting more pressure on the data platforms underneath. This leads to a greater demand for observability architectures that can manage more data, more users, and more machine-to-machine interactions without losing speed.

For Tschetter, the solution is not to create new observability tools, but to rethink the data layer that supports them.

What is Imply’s ‘Observability Warehouse’?

A key point during the podcast was what Imply refers to as the “observability warehouse.” This idea applies modern concepts from data lakes and warehouses to observability and security workloads. 

Instead of making enterprises move data between specialised platforms, Tschetter recommends storing data once and allowing different teams and tools to access it in their preferred ways.

“The true way to break down silos is to store the data once and access it from multiple different places,” he told Petrie.

This idea also applies to how enterprises handle log data. Tschetter challenged the notion that logs and structured data are fundamentally different. In reality, he argued, the difference is less about the data itself and more about how users want to query it.

“The actual structure of the data is not a function of the data itself,” he explained. “It’s a function of how people want to interact with the data.”

What is Imply Lumi Loglake?

Imply’s Chief Architect’s perspective aligns with Imply’s vision of a “Loglake.” The layer sits on top of existing data storage and presents information in a way that is familiar to observability and security teams. Whether users rely on SPL, KQL, LogQL, or other query languages, they can keep using their current workflows while accessing the same data underneath.

Such an approach reflects a larger trend toward separate architectures. Drawing parallels with the development of business intelligence (BI), Tschetter noted that analytics platforms gradually separated data storage from visualisation and reporting tools. He thinks observability is moving in the same direction.

“Different teams want to query the same data, but they want to do it differently,” he said.

This need is especially clear in the growing overlap between observability and security operations. While these two areas often have different leadership structures, Tschetter sees considerable similarities in both the data they use and the questions they ask.

Observability teams look at logs to understand application performance and reliability. Security teams examine many of the same records to investigate breaches, suspicious activity, and compliance issues. Often, both groups rely on the same datasets.

What’s the difference between Observability and Security Teams’ Strategies?

The key difference in the observability team’s approach and the security team's approach is in their time frames. Site reliability engineers may look into incidents from the past week, while security analysts may need to follow activity going back several years. Keeping that historical visibility can become costly within traditional Security Information and Event Management (SIEM) platforms.

Tschetter believes that separating the data layer from the analytics layer offers a more practical solution. SIEM tools can continue detecting and investigating threats while a specialised observability warehouse provides cost-effective storage and computing resources.

Are you enjoying the content so far?

“Our focus on just the data layer allows us to provide a great unit cost in terms of storage and compute,” he said. “Users don’t even have to know that we exist because they’re interacting with the SIEM, not necessarily with the data.”

As AI adoption speeds up, Tschetter thinks organisations should put less emphasis on connecting more systems through agents and more on creating a shared, scalable data foundation. He believes the future of observability, security, and analytics relies on a common architecture that allows multiple teams to work from a single source of truth.

For CIOs, CISOs, and chief data officers, the message is clear: organisations that remove data silos, rather than just connecting them, will be better positioned for the AI era.

Key Takeaways

  • AI is transforming observability and security disciplines.
  • The observability warehouse concept is gaining traction.
  • AI agents increase the volume of queries significantly.
  • Data silos remain a major challenge for enterprises.
  • Collaboration between IT and security teams is essential.
  • Observability and security teams often consume the same data.
  • A decoupled architecture can enhance data accessibility.
  • The semantic layer must support multiple query languages.
  • Effective data management is crucial for AI-driven workloads.
  • Data should be stored once and accessed from multiple platforms.

Chapters

  • 00:00 Introduction to AI and Observability
  • 02:08 Challenges in Observability with AI
  • 06:44 Modernising Architecture for Observability
  • 10:49 Decoupled Observability and Semantic Layers
  • 16:31 Collaboration Between IT and Security Teams
  • 22:23 Imply's Observability Warehouse and Data Lakes

For more information on AI, observability and Imply’s observability warehouse and data lakes, please visit imply.io.

For further information on all things B2B Tech, please visit em360tech.com

Imply LinkedIn: @Imply 

Imply X: @implydata

Imply YouTube: @Implydata

EM360Tech YouTube: @enterprisemanagement360

EM360Tech LinkedIn: @EM360Tech

EM360Tech X: @EM360Tech

Follow: @EM360Tech on YouTube, LinkedIn and X

Stay connected for more expert insights, podcast episodes, and enterprise data strategy discussions.