Member-only story
Data Engineering is Dead?
I keep hearing: “AI needs data, so learn data engineering. Without data there’s no AI.” True — but incomplete. Let’s be honest: some folks say this because they sell courses (I create courses too — but it’s not my main income, so I’ll be candid).
Most companies don’t build foundation models; they consume them. So the average data engineer isn’t training LLMs. We’re still enabling analytics, BI, and ML — but how we do that work is changing.
With today’s LLMs, you can prompt ChatGPT to scaffold a notebook that ingests from a data lake, performs a merge/upsert, and writes to a Bronze table — then ask it to add tests and basic anomaly checks. It’s not perfect, but it’s a strong head start. Agentic, AI-assisted ETL tools are appearing that nudge you to clarify requirements and fill gaps before generating pipelines.
What does that mean? Saying it’ll take two weeks to hand-code a basic landing-to-Bronze pipeline won’t fly for long — an AI-savvy analyst could co-pilot a workable scaffold in hours. Development costs will drop. The value shifts to problem framing, validation, and governance. Many data engineers will look more like AI-aware solutions architects, and there’s growing opportunity around unstructured data (text, audio, video) where pipelines are still immature.
