I Had a Colleague Ask Me How to Do Incremental Load on a Complex Reporting Table — Here’s What I Told Him

Photo by Gabrielle Henderson on Unsplash

One of the most common performance pain points I see in data pipelines is this:

“We have a table with heavy transformations and business logic, and our full refresh process is painfully slow. How do we make it incremental?”

Recently, a colleague ran into this exact problem.
He was responsible for a reporting table powering executive dashboards — but the nightly full refresh was taking hours, eating up cloud compute, and still occasionally failing before business hours.

He asked me, “How do I load this incrementally without breaking all the logic and joins?”

Here’s how I helped him fix it — and what you can take away if you’re facing something similar.

💥 The Challenge

The reporting table was derived from:

• A wide fact table with transactional data
• Joins to several dimension tables for enrichment
• A long list of business rules involving filters, mappings, aggregations
• And finally, a write to a Delta Lake table used for BI dashboards