AI (Artificial Intelligence) and GenAI (Generative Artificial Intelligence) are rapidly reshaping how industries approach automation, data analysis, and productivity. AI is often associated with ML (machine learning) algorithms that automatically extract valuable insights from data. When it comes to GenAI, applications like ChatGPT for conversational AI and DALL·E for image generation are among the first that come to mind. However, the fields of application are more widespread. For example, if you are a developer or involved in coding in any form, chances are you have already encountered GenAI as a tool for assisting with code generation and optimization.
One area where GenAI is beginning to show potential is data engineering. In this article, we will illustrate the use of AI in this area. Data engineering involves designing, constructing, and maintaining data pipelines that enable businesses to collect and store data efficiently. It is, therefore, a crucial prerequisite for organizations aiming to leverage data for ML, analytics, and business intelligence.
Example use cases
AI can play a role in optimizing data engineering tasks, making them more efficient and scalable. Below are examples of key areas where AI can significantly contribute:
Code Generation
Since creating code, be it standalone scripts or production-ready software, is an integral part of data engineering, GenAI capabilities in code generation can be useful. This means that a data engineer can, e.g., describe what shall be accomplished in natural language, and an AI assistant will create the first draft of the code. Alternatively, they can ask for help if there are problems with their code. Of course, the data engineer needs to understand the code to ensure it is correct and to avoid creating a black box application that lives a shadow life outside of any human control. Importantly, the power of GenAI code generation is limited; it will likely struggle with complicated or novel approaches. Reasoning-Models, i.e., models that make a decision while following a coherent line of thought, are better equipped for code generation than models without reasoning capabilities. Nevertheless, it is essential to adjust the expectations for AI to be a limited assistant, not a fully automated programmer. This disclaimer holds for all the following examples.
Data Pipeline Automation
AI-powered tools can support the creation of basic ETL/ELT (extract, transform, load) scripts in languages like SQL and Python, reducing the manual effort required to build and maintain data pipelines. These tools can also analyze existing workflows and suggest or generate code for data extraction, transformation, and loading. For example, SQL queries for data extraction can be generated from natural language.
Query Optimization
AI can analyze query patterns, detect inefficiencies, and recommend optimizations to improve performance. Machine learning models can predict optimal query execution plans, reducing latency and resource consumption in large-scale databases.
Synthetic Data Generation
Access to valuable data can, and in many cases should, be heavily restricted. For example, sensitive information such as PII (personally identifiable information) data must be protected from unauthorized access, and often, real-life data is simply not yet available during development. Nevertheless, data engineers need to build data pipelines, which is difficult without data to base them on and test them with.
Given that the underlying information is known, AI can also generate synthetic datasets that mimic real-world structures and distributions. This is particularly useful in scenarios where data privacy is a concern or when dealing with incomplete datasets. Synthetic data enables testing machine learning models, improving their robustness without exposing sensitive information.
Data Labeling
Labeling large datasets is time-consuming, but it is often necessary, especially for supervised learning models. AI-driven labeling tools can automatically tag and classify data, significantly accelerating the process. These models should be monitored and refined over time with human-in-the-loop validation. The AI model can also find edge cases that a human should check.
In a previous project at CID years before the current AI hype, we automatically detected events in news articles. Since no readymade training data was available for our specific event definitions, we had to do the labeling ourselves. We used the current classifier’s output to find the most promising documents for further labeling result improvement. Given current GenAI abilities, much of this manual human labeling work can be omitted.
Schema Mapping
AI can analyze and unify disparate schemas from different databases by detecting similarities and suggesting appropriate mappings. For example, common names for fields that are likely to contain the same information (e.g., “name” & “full_name” for person data, or “{‘price’: x, ‘currency’: ‘USD’}” & “price_usd” for product prices) can be mapped automatically. This is crucial in data integration projects where organizations need to merge datasets from various sources while maintaining consistency and integrity. However, like in the code generation use case, an AI system cannot magically solve this problem in all cases. It will also fail if the data and schema description do not contain enough information for a human to create a precise mapping. For instance, if the price of an item has no currency information attached to it, an AI might assume that the currency is USD since it is the most likely guess, which might be wrong if the data is about the European market.
Limitations of AI in Data Engineering
Accuracy and Manual Review
Despite advancements, AI-generated outputs are not always 100% accurate. Human oversight is necessary to validate results, especially in critical data engineering tasks where errors can have significant downstream effects.
Even a seemingly straightforward task, such as the schema mapping example we mentioned earlier, can get complicated, especially if the needed context is missing. An AI system might be quick to assume that “customer_id” always refers to the same real-life thing, but depending on the context, it might be an end customer or a distributor. Moreover, naming conventions can be different across systems; the “customer_id” in one system might only be “id” in a second one, while the “client_id” in this second system refers to the “customer_id” in the first one, etc. These differences are already hard for a human to grasp, but one needs to be especially careful about providing the necessary context to an AI system and monitor the system’s output to avoid mistakes.
Undetected errors in AI output can quickly lead to data corruption throughout the pipeline, rendering carefully curated metadata useless and causing a loss of trust in the overall data quality.
Data Bias
AI models learn from historical data, which may contain biases. For example, if a system that monitors data quality and alerts in case of issues is trained on a biased subset of the data (e.g., only representation of one product group), it will fail to generalize to other subsets of the data. If not addressed, these biases can lead to incorrect alerts and missing alerts for potential data quality issues. Ensuring diverse and representative training data is essential to mitigate this risk.
A special issue arises if a system based on biased training data is used further in the training of another system, reinforcing the bias. This is another reason why it is important to detect and eliminate biases as soon as possible in a pipeline.
Interpretability
AI-driven decision-making processes are sometimes opaque. Especially deep-learning-based ML models, such as LLMs (large language models) used in GenAI applications, struggle with explainability, while this is less of an issue for many more traditional ML approaches, e.g. decision trees or linear regression. Combined with imperfect accuracy, missing interpretability and transparency can lead to unreasonable debugging efforts. If something goes wrong, an engineer might have to guess why the AI did what it did and might be unable to fix it, requiring a workaround. Errors might even go unnoticed. Understanding how AI arrives at specific conclusions is vital, especially in regulatory environments where transparency is needed.
Security and Compliance
We have already mentioned that missing interpretability can increase compliance risk, especially in highly regulated industries such as finance or healthcare. If an audit requires a transparent view of the data lineage, an AI black box can become a huge risk. All AI-generated data workflows must comply with industry standards and regulations, such as GDPR or the EU AI Act.
Importantly, access to sensitive data needs to be restricted. While this may seem obvious in certain cases, such as avoiding the careless use of PII in AI tools that are not fully under your control, risks can arise much earlier in the development process. Even during the initial stages of building a data pipeline, sensitive information can inadvertently be exposed. Navigating this requires a careful touch and a strong awareness of data sensitivity to determine what needs protection and to what extent. Ensuring AI tools align with security best practices is essential to avoid data breaches and legal issues.
Conclusion and Future Outlook
While AI is starting to play a meaningful role in data engineering by helping streamline certain tasks, gaining insights faster, and supporting new methods like synthetic data generation, it is no universal, quick solution for everything. Real challenges remain around data quality, bias, model transparency, and regulatory constraints. As AI tools become more integrated into data workflows, balancing innovation with a clear view of their limitations and risks is important.
AI’s impact on data engineering will likely expand in the coming years. Now is the time for organizations to build the foundations: invest in data quality, strengthen governance, and experiment with AI tools in controlled, well-understood environments. Responsible adoption today will create the flexibility to adapt as the technology matures.
Are you excited to get started? Contact us to assess your situation. We look forward to finding a path forward with a measured, pragmatic approach to AI adoption.
