Why I dislike Delta Live Tables

Delta Live Tables are an innovative technology that enables real-time processing and analysis of big data.

According to Databricks

“Delta Live Tables (DLT) makes it easy to build and manage reliable batch and streaming data pipelines that deliver high-quality data on the Databricks Lakehouse Platform. DLT helps data engineering teams simplify ETL development and management with declarative pipeline development, automatic data testing, and deep visibility for monitoring and recovery.”

Databricks is attempting to gain some share in the ETL orchestration, data lineage, quality and discovery market which is commendable. However, I just think they rushed to deliver without fine-tuning the product for easy adoption.

Despite its advantages, Delta Live Tables may not be the best option for every use case, and here are the top reasons why:

1. We can’t run the Delta Live tables notebook in debug mode currently. You can’t import Dlt. This is the most basic thing a developer needs to be able to do. However, it is not possible with Delta Live tables currently, which is frustrating. The only workaround is that a developer has to test their code from the workflow menu in Databricks to see if it works.

2. A developer can’t customize their target delta table file paths to their desired location for bronze, silver or gold tables. Normally, in the real world we have different storage containers to store bronze, silver or gold tables. However, with Delta live tables it stores them in one container.

3. Limited integration: Delta Live Tables are not currently integrated with a wide range of data sources and tools, which means that organizations may need to use workarounds or other solutions to access their data. Popular ETL and orchestration tools like Data Factory and Airflow don’t integrate seamlessly with Delta Live tables.

4. Cost: Delta Live Tables are designed for large scale data processing and therefore require substantial computing resources to run effectively. This means that the cost of using Delta Live Tables can be significant, especially for smaller organizations.

5. Complexity: Delta Live Tables have a relatively steep learning curve, and their complexity can make them…