There’s been a lot of excitement recently around agentic AI in data engineering, particularly around automating (or even “replacing”) parts of the ingestion and transformation layer. The promise sounds great, spend less time on repetitive engineering work, more time on business value, insight, and storytelling.
At Coeo, we think we’re at a fork in the road. One path leads to faster, smarter, more consistent engineering. The other? An AI-generated mess.
What gives us the right to have an opinion here? We have real deployment experience of enterprise-level complexity. For one major UK retail client, we used some of the techniques below to save them thousands of hours of development and deliver a successful project where others had failed.
Trivialising Engineering
There is a push to move data engineers away from highly repeatable tasks, such as routine data ingestion, toward more business-centric, value-driven areas, such as data modelling and storytelling. In principle, that’s a good thing.
The challenge is how we define “routine.”
In most enterprise environments, ingestion from structured sources like SQL Server is already largely solved. Many teams run metadata-driven pipelines that handle thousands of tables, support incremental load patterns (change tracking, date-based deltas), and scale across complex source systems. It’s not cutting-edge anymore, but it’s not trivial either.
Modern platforms like Microsoft Fabric are trying to simplify this with features like Mirroring and Copy Jobs. I’m a fan of where they’re headed, but in practice, these tools struggle at true enterprise scale. Mirroring is conceptually powerful but comes with enough limitations that it’s hard to apply beyond specific use cases. Copy Jobs work well for small numbers of tables or modest data volumes, but don’t yet offer the performance tuning, observability, or governance you get with a full pipeline-driven approach.
This is where I get sceptical about agentic AI being positioned as a “replacement” for data engineers. If AI simply generates ingestion logic without strong structure, standards, or validation, you just get a faster way to produce inconsistency.
The Reality
We’ve been here before.
The tools evolve, from DTS to SSIS to ADF, and now to Databricks, Fabric and beyond, but the role of the data engineer doesn’t disappear. It shifts. The value moves up the stack: architecture, data modelling, governance, performance design, and ultimately, how data is shaped into something the business can trust and use.
Agentic AI feels like the next tool on the shelf. For it to work in real-world, regulated, enterprise environments, it still needs strong developer oversight and governance. Supporting the “happy path” is easy. Supporting semi-complex, messy, exception-heavy scenarios? That’s where real engineering lives.
And then there’s the areas where AI still struggles, non-routine ingestion. Excel files that change shape every week. “Business logic” that lives in someone’s head instead of a specification. AI might get better at this, but for now, it’s one of the clearest areas where human engineers add disproportionate value.
AI as a Force Multiplier
Where my thinking really changed was seeing agentic AI used not to invent solutions from scratch, but to operate within frameworks that engineers have already designed.
Instead of asking an agent “Build me a pipeline,” you ask: “Use this framework, follow these standards, validate your output, and generate a job definition that fits our architecture.”
In that model, AI becomes an accelerator for good engineering rather than a source of randomness. It can remove much of the friction around “How do I use this framework?” and “What’s the right pattern for this case?”, especially for newer team members or fast-moving projects.
I’m genuinely excited about using an agent to generate job definitions for a proven ingestion or transformation framework. If it can consistently apply naming conventions, patterns, validation rules, and documentation standards, that’s a huge win for both speed and consistency across a platform.
Automation to Augmentation
For me, the real value of agentic AI in data engineering isn’t about replacing engineers. It’s about amplifying them.
The future I want to see is one where AI handles the repetitive, structured, rules-based work inside well-defined guardrails, and engineers spend more time designing those guardrails, shaping data models, and working with the business to turn raw data into something meaningful.
The risk is building systems that let AI run free and calling it “automation.” The opportunity is to build systems where AI becomes a first-class citizen within strong engineering frameworks.
One leads to an AI-generated mess.
The other leads to better engineering at scale.
And I think the difference between the two is entirely in how we choose to design for it today.
If you want to learn how to accelerate your projects with the right combination of AI and mature tooling & deliver value back to the business quicker than ever, reach out to your Coeo Account manager or reach the team at info@coeo.com.