While reviewing the Databricks documentation about Unity Catalog, I came across a concept that initially seemed a bit perplexing: the distinction between accessing data objects stored in our cloud storage using External Tables versus External Volumes. This inspired me to write an article exploring the different methods for accessing data…

Large Language Models (LLMs) like GPT-4 can indeed be applied creatively to address various challenges in the field of data engineering. Here are some ways LLMs can be leveraged to solve data engineering challenges: Data Pipeline Documentation: Writing and maintaining documentation for complex data pipelines can be time-consuming. LLMs can…

While data engineering is a rewarding and in-demand career for many, there are certain types of individuals who might find the role challenging or less suitable for their strengths and preferences. Here are some scenarios where someone might consider not pursuing a career as a data engineer: Limited Interest in…

Recently, I had a discussion with a colleague about deployment strategies for data engineering teams. We were comparing the trunk-based approach to the branch per environment strategy. I’ve noticed that the trunk-based approach is recommended by Microsoft and I’ve also witnessed its implementation in a project. Additionally, I’ve seen the…

Introduction: Data modeling is the foundation of effective database design, enabling organizations to structure and represent their data for efficient storage, retrieval, and analysis. …

Introduction: In the era of data-driven decision-making, time series data has emerged as a critical asset for businesses across various sectors. With the evolution of data architecture, storing and analyzing time series data in a data lakehouse platform presents both opportunities and challenges. In this blog post, we’ll explore the best practices for effectively managing time series data in a data lakehouse, delve into the challenges you might encounter, and provide practical solutions to overcome them.

In the modern landscape of data-driven decision-making, data pipelines have emerged as the lifeblood of organizations. They facilitate the seamless flow of information, enabling businesses to extract valuable insights and drive growth. However, amid the pursuit of data-driven success, there’s a crucial aspect that demands our attention: the measurement of…

In the ever-evolving landscape of data management, the question persists: Is there a truly exceptional Change Data Capture (CDC) tool available in today’s market? This query has resonated with me throughout my career, prompting me to explore various CDC solutions. Along the way, I’ve engaged with industry stalwarts such as…

In the realm of data engineering, challenges often present themselves in the most unexpected ways. Recently, I embarked on a journey with Databricks that led me to an intricate puzzle — one that took days of exploration and experimentation to crack: the enigmatic non-idempotency of the Row_Number() window function. My…