Databricks offers various methods to create copies of existing Delta Lake tables, each serving specific use cases. Let’s explore the characteristics and use cases of shallow clones, deep clones, views, and the CREATE TABLE AS (CTAS) approach. 1. Shallow Clones: Definition: Shallow clones create a snapshot of a Delta table without copying the…

In this blog post, my aim is to guide you on how to effectively utilize the Unity Catalog within your current data lakehouse and offer my recommended best practices. These practices are designed to help you fully leverage the new features of the Unity Catalog without causing disruptions to your…

The motivation behind writing this article is to explore the potential cost savings associated with Databricks job clusters and to determine whether using them within Data Factory or in Databricks workflows is the better option. As we are aware, all-purpose clusters tend to be more expensive than job clusters. This…

This article was inspired by a real-world underperforming Spark Streaming pipeline handling IoT data. One of my initial investigative steps was to identify the root cause of performance bottlenecks during upserts or inserts into a delta table. It became evident when I attempted to read that Delta table and create…

In the fast-evolving landscape of big data processing, choosing the right file format can significantly impact your data pipeline’s efficiency and effectiveness. Parquet and Delta are two popular formats, each with its own strengths and ideal use cases. In this article, we’ll explore when it makes sense to opt for…

In the realm of Data Engineering, the methods we use to extract data are just as pivotal as the data itself. Two primary strategies, known as Pull and Push, stand at the forefront of shaping how we collect and process information. …

Introduction: — In our fast-paced digital world, mastering data engineering is essential for personal and professional growth. The good news is that there are various pathways to achieve this mastery. In this blog post, we’ll explore and compare three popular approaches: self-paced learning, master’s degree programs, and bootcamps. …

While reviewing the Databricks documentation about Unity Catalog, I came across a concept that initially seemed a bit perplexing: the distinction between accessing data objects stored in our cloud storage using External Tables versus External Volumes. This inspired me to write an article exploring the different methods for accessing data…

Large Language Models (LLMs) like GPT-4 can indeed be applied creatively to address various challenges in the field of data engineering. Here are some ways LLMs can be leveraged to solve data engineering challenges: Data Pipeline Documentation: Writing and maintaining documentation for complex data pipelines can be time-consuming. LLMs can…