Data Engineering Is Moving From Writing Pipelines to Proving Intent

For years, data engineering has largely been defined by implementation. Engineers take a business requirement, turn it into technical logic, build the pipeline and make sure data reaches the right place in a usable form. AI has already made parts of that process faster. Now, agentic tooling is starting to change the process itself. 

Google's Data Engineering Agent can create and modify orchestration pipelines from natural-language instructions, change schedules, investigate failures and suggest fixes. Databricks' Genie Code can plan multi-step data engineering work, retrieve relevant assets, generate and run pipeline code, then respond to the results. 

These aren't just copilots finishing lines of SQL. They're beginning to participate across the pipeline lifecycle. That doesn't mean data engineers are about to stop writing code. Current tools still have important limitations, and Google's own agent remains a preview product that requires humans to review and run generated pipelines. 

But the direction is becoming clearer. As implementation becomes easier to automate, more of the difficult engineering work moves elsewhere. The question is no longer simply whether an AI agent can build a working pipeline. It's whether the organisation can tell the agent exactly what the pipeline needs to achieve, then prove that what it built is actually correct.

AI Is Changing the Abstraction Layer of Data Engineering

There's a difference between making an existing task faster and changing the level at which people perform it. Traditional coding assistants mostly do the first. An engineer still decides how a pipeline should be implemented, but AI helps write the SQL, Python or configuration required to make it happen. The implementation model stays largely intact. 

The person is simply producing it faster. Agentic data engineering pushes the abstraction higher. Google's Data Engineering Agent, for example, accepts natural-language requests to create or change an orchestration pipeline. It generates a plan for review, performs the requested work and then verifies the result using tests or dry runs. 

Databricks similarly describes Genie Code's Agent mode as capable of planning, generating and fixing entire pipelines through multi-step workflows. This builds on a change data platforms were already making. 

Declarative data pipelines let engineers describe more of the state or outcome they want while the underlying platform works out some of the execution mechanics. Agentic systems take another step by allowing more of those requirements to be expressed through natural language and context. The progression starts to look something like this:

Code → Generated code → Declarative configuration → Intent + constraints → Verification

We shouldn't read that as a fixed maturity model every data team will follow. Nor does each stage replace everything before it. Engineers using agentic tools will still work with code, configuration and orchestration systems. What changes is where more of their attention goes. 

Once a machine can produce a larger share of the implementation, writing every instruction manually becomes less of the bottleneck. Accurately expressing what the implementation is supposed to accomplish becomes much more important. And that's harder than it sounds.

Intent Is Not an Engineering Specification

Imagine asking an agent to build a daily pipeline that combines customer transactions and produces a revenue table. The desired outcome seems fairly clear. But:

• What exactly counts as revenue?
• Which system is authoritative if two sources disagree?
• Should refunded transactions remain in the dataset?
• How fresh does the data need to be?
• What happens when a customer record is duplicated or an upstream schema changes?

Those aren't small implementation details. They're part of what determines whether the final data product is correct. A human engineer can encounter these questions while building the pipeline and resolve them using documentation, conversations, existing code and knowledge accumulated over years of working with the environment. 

Some of that knowledge may never have been formally documented because the engineer already knows it. That creates a problem for AI-generated pipelines. Natural language can communicate an objective without carrying all the assumptions hidden behind it. If an agent is expected to make more implementation decisions, those assumptions have to become visible somewhere else. 

Google's own guidance gives us an early indication of what this looks like. Its Data Engineering Agent supports reusable instruction files containing rules such as naming conventions, data layers and conditions governing how pipelines should behave. Google also recommends providing the agent with relevant files and context because its performance depends partly on what it can see. 

This is why prompt engineering is too shallow a way to describe the shift. Data engineers don't just need to become better at asking an AI tool for something. They need to turn tacit technical and business knowledge into data pipeline requirements that a machine can interpret consistently. 

The quality of the specification starts becoming part of the engineering itself.

Constraints Turn Intent Into Something an Agent Can Engineer

Intent tells an agent where the organisation wants to go. Constraints define the conditions it has to respect while getting there. That distinction becomes increasingly important as pipeline automation takes on more implementation work. 

A generated pipeline can satisfy the broad request while still violating assumptions that would have been obvious to the people who normally build or operate it. For enterprise teams, those constraints generally fall into four connected areas.

Data constraints define what must remain true

An agent needs more than access to the source and destination. It needs to know the structural and quality conditions the resulting data has to preserve. That can include expected schemas, record grain, freshness thresholds and which sources should be treated as authoritative. 

Data contracts can help formalise some of these expectations by defining what producers and consumers can rely on as data moves between systems. Making these conditions explicit also gives teams something concrete to validate when a pipeline changes. 

The question stops being whether the agent successfully moved the data and becomes whether the resulting dataset still satisfies the conditions its consumers depend on.

Business constraints define what correct actually means

Technical validity doesn't guarantee business accuracy. A pipeline might run perfectly while applying the wrong definition of an active customer, recurring revenue or completed transaction. If those definitions live only inside dashboards, documentation or someone's head, an AI agent can produce technically sound logic around the wrong interpretation. 

So business meaning has to become part of the specification too. Business logic, transformation rules and shared definitions need to be clear enough for both people and machines to apply them consistently. This is where data engineering increasingly overlaps with semantic discipline. 

Teams need a dependable way to express what the organisation means by its data, not just where that data sits.

Operational constraints define acceptable behaviour

A pipeline is also defined by what happens when things don't go according to plan. An upstream dependency might arrive late. A schema could change. A transformation might suddenly process far more data than expected. One service may become unavailable while the rest of the workflow continues normally. 

Reliable data orchestration already accounts for dependencies, retries and recovery. Agentic engineering doesn't remove those requirements. Instead, it makes it more important that acceptable behaviour is clearly defined before an automated system starts making implementation or repair decisions. 

The agent needs boundaries for what it can fix, what it can retry and when a problem needs a person.

Governance constraints define what the pipeline is allowed to do

The same principle applies to governance. An agent may be technically capable of joining two datasets, creating a new output or changing how information is processed. That doesn't automatically mean the organisation permits it. 

Access rules, sensitive-data handling, retention requirements and lineage expectations have to survive data engineering automation just as they do manual development. Governance becomes another constraint the implementation has to satisfy rather than something teams apply after the pipeline has already been generated. 

Making these four areas explicit gets an organisation much closer to a usable engineering specification. But specification only establishes what should happen. Teams still need evidence that it did.

A Working Pipeline Still Has to Prove It Is the Right Pipeline

Code can compile. A pipeline can run. Data can appear in the target table. None of those things proves the result is correct. Snowflake AI Research demonstrated the difference in August 2026 with data-eng-bench, an open benchmark designed specifically for repository-level data engineering. 

Its 103 tasks require agents to build and repair pipelines against an enterprise-scale warehouse rather than solve isolated SQL exercises. The interesting part is how those pipelines are judged. 

The benchmark checks hidden assertions covering factors such as output grain, column contracts, formula correctness, edge cases and whether pipelines produce consistent results when rerun. An agent doesn't pass because its SQL looks sensible. Every required assertion has to hold. Even then, reliability varies. 

Snowflake reported that its strongest tested configuration solved 73.8 per cent of tasks on the first attempt, but only 64.1 per cent were solved across all three independent runs. The benchmark comes from Snowflake and Bespoke Labs, so it shouldn't be treated as a neutral verdict on every agentic platform. But it illustrates the engineering problem clearly. 

Data pipeline validation has to test the conditions the business actually cares about, not simply whether generated code executes. The same shift is appearing beyond data engineering. GitLab's 2026 AI Accountability Report, based on 1,528 DevSecOps professionals across six countries, found that 85 per cent agreed AI had moved the software development bottleneck from writing code towards reviewing and validating it. 

Data teams could face a similar trade-off. Generating implementation faster doesn't automatically make the whole engineering process faster if the organisation can't efficiently establish whether that implementation is safe to trust. Verification therefore isn't the final check after agentic engineering. 

It's one of the conditions that makes greater automation possible in the first place.

The Data Engineer's Job Moves Towards Judgement, Not Prompting

The obvious conclusion would be that data engineers write less code and spend more time telling agents what to do. Reality is more complicated. Astronomer's State of Airflow 2026 survey, which gathered responses from more than 5,800 data practitioners across 122 countries, found that more than 80 per cent already use AI to write Airflow pipelines. 

Yet generic AI tools still struggle with the specialised context of the job. Forty-three per cent cited hallucinations as a major issue, 42 per cent reported outdated syntax and just nine per cent said they were satisfied with DAGs generated by generic AI tools. So the data engineer role isn't simply turning into prompt engineering. 

Someone still needs enough implementation knowledge to recognise a poor plan, understand why a transformation is wrong and judge whether an automated fix introduces a new problem somewhere else. What does change is the centre of gravity. 

More value starts to sit in defining what must be true before work begins, identifying where automation needs boundaries and deciding what evidence is strong enough to accept the result. Those responsibilities require knowledge of the data environment, the platform and the business behind it. 

They also expose a weakness many organisations already have. Important engineering knowledge is often scattered across code, tickets, documentation and individual experience. Humans have traditionally been good at filling those gaps informally. Agents need that context made much more explicit. 

For data engineering leaders, preparing for agentic tooling may therefore involve more than choosing which platform has the strongest agent. It also means asking whether the organisation has captured enough of its own engineering judgement for automation to use reliably.

What Data Leaders Need to Ask Before Scaling Agentic Data Engineering

There is a clear economic reason to automate more pipeline work. Fivetran's 2026 Enterprise Data Infrastructure Benchmark, based on more than 500 senior data and technology leaders at large organisations, found that respondents estimated 53 per cent of engineering time was being spent maintaining pipelines. 

Fivetran has a commercial interest in automated data integration, but the scale of the reported maintenance burden helps explain why vendors and data teams are pushing towards greater automation. The harder question is whether the organisation is ready to delegate more of the work without losing control over the result. 

Before scaling agentic data engineering, data leaders should be able to answer four questions:

• Can we express the intended outcome precisely enough for an agent to act on it? If different engineers would interpret the same request differently, automation is likely to inherit that ambiguity.
• Are the critical constraints explicit? Business definitions, source assumptions, data quality rules, operational boundaries and governance requirements can't depend entirely on undocumented team knowledge.
• Can we automatically verify the result? Tests need to prove that generated pipelines satisfy the conditions that define correctness, not only that the code runs.
• Do we know who owns the decision when technically valid output is still wrong? Automation can perform work, but accountability for accepting the result still needs a clear home.

These questions give teams a more useful measure of AI readiness than simply counting how many pipeline tasks an agent can automate. The more implementation an organisation wants to delegate, the more clearly it needs to define the conditions under which that implementation can be trusted.

Final Thoughts: Data Engineering Still Owns the Definition of Correct

Agentic tools are making it possible to build, modify and troubleshoot data pipelines at a higher level of abstraction. As those capabilities mature, engineers may spend less time translating every requirement manually into code and configuration. But less implementation doesn't mean less engineering. 

Intent describes the outcome. Constraints define the boundaries around it. Verification provides evidence that the implementation has actually delivered what the organisation intended. That leaves human judgement at the centre of the future of data engineering, even as more execution moves to AI agents. 

The role shifts towards making assumptions visible, turning organisational knowledge into usable specifications and deciding when the evidence is strong enough to trust what automation has produced. Organisations that do this well won't simply have better prompts. They'll have a clearer definition of what good data engineering looks like, expressed in a form both humans and machines can work with. 

As agentic data platforms continue to mature, EM360Tech will keep examining where those changes move responsibility across enterprise data teams, and what leaders need to put in place before automation moves from useful assistant to operational dependency.