Key Takeaway
Chris highlights a shift towards more agentic data engineering, where AI agents handle operational tasks, allowing data engineers to focus on strategic goals and overall data management. This evolution could lead to data engineers taking on more strategic roles within organizations. Recent data shows that 74% of teams have seen improved productivity and 77% report enhanced output quality due to AI integration. Dave notes that advancements in generative AI have provided data engineers with powerful tools for automating tasks like data cleansing and workflow orchestration. Consequently, data engineers must also develop business acumen and communication skills to maximize organizational value.
Chris explains: “We’ll begin to see more agentic data engineering, where AI agents handle a larger portion of operational tasks, enabling data engineers and teams to focus on the bigger picture.
“They’ll ask, ‘What are our overarching goals? How should we allocate budget to these agents for data processing? How do we consider our overall data estate instead of just individual pipelines?’
“This will lead to a significant shift in the role of data engineers.”
This transformation means that data engineers could evolve within organizations to assume more strategic roles.
AI-powered data engineering
Over the past two years, 74% of respondents reported enhanced productivity in data engineering teams due to AI, reflected in the volume of work completed.
Quality has also seen a significant uptick, with 77% noting a marked improvement in the team’s output.
Dave states: “With advancements in Gen AI over the last two to three years, and crucially, its integration into software development tools, data engineers now possess a powerful accelerator at their disposal.”
AI-powered tools now automate and assist data engineers with tasks such as data cleansing, integration, pipeline monitoring, metadata management, workflow orchestration, feature engineering, and other essential functions.
The research indicates that today’s data engineers must not only master AI but also cultivate business acumen, communication, and presentation skills to provide genuine organizational value.
