Published inGeek Culture·PinnedMy Guide to Becoming a “Market-Ready” Data Engineer With a $100 Investment in 3 Months or Less — 2021As you may know, the demand for Data Engineers has quadrupled in the last 2 years. There are more jobs than candidates in the job market, especially for US work authorized candidates. I have seen companies offering up to $1500 in referral bonuses. My goal in writing this post is…Data Engineering10 min readData Engineering10 min read
Mar 25Member-onlySQL NULLs: Facts that might Shock YouHandling SQL Nulls effectively and writing bug free code is something I am yet to master, even after writing SQL for over 8 years. In this blog post, I will expose some of the facts about Nulls that I didn’t even know and share some of the common coding mistakes…Data5 min readData5 min read
Feb 24Member-onlyMy Review of Databricks Delta Lake Change Data FeedIn this article, I will give my unbiased assessment of Databricks Delta Lake change data feed, analyzing the potential use cases, pros and cons. One of the recent features introduced by Databricks is Delta Table Change Data feed. Databricks describes it as below “Change data feed allows Databricks to track…Data3 min readData3 min read
Feb 6Member-onlyWhy I dislike Delta Live TablesDelta Live Tables are an innovative technology that enables real-time processing and analysis of big data. According to Databricks “Delta Live Tables (DLT) makes it easy to build and manage reliable batch and streaming data pipelines that deliver high-quality data on the Databricks Lakehouse Platform. DLT helps data engineering teams…Data Analytics3 min readData Analytics3 min read
Published inGeek Culture·Jan 18Member-onlyDesigning Incremental data extraction for a source data that is a SQL queryThe goal of this article is to describe how we can design and implement an incremental ETL pipeline that extracts data from a source query (instead of the common case of a table) and load it to a Target table. Often times, most people are knowledgeable in designing incremental data…Data Engineering5 min readData Engineering5 min read
Aug 17, 2022Member-onlyTime Travel versus Slowly Changing Dimension Type 2The motivation for writing this article is to help readers realize and recognize the opportunity to utilize the Delta Lake time travel feature when faced with a problem that involves analyzing historical versions of a record (slowly changing dimension type 2). …Data Engineering5 min readData Engineering5 min read
Published inGeek Culture·Jul 12, 2022Member-onlyCracking the Data Engineering Interview — Part 2: GuideData engineering interviews can be quite frustrating especially when you get a lot of recruiter calls, and interview request that all end up with a rejection letter. In the last post, I described the typical data engineering process and offered some specific tips for each stage. …Data6 min readData6 min read
Published inGeek Culture·Jul 7, 2022Member-onlyCracking the Data Engineering Interview — Part 1: StructureThe goal of this blog post is to offer some general information about the data engineering interview process and offer some useful tips that might help a newbie land their first data engineering job. As a senior data engineer that has transitioned from a completely non-IT-related field into data engineering…Data4 min readData4 min read
Published inGeek Culture·May 3, 2022Member-onlyBeginner's Guide: Development Life Cycle in Data EngineeringThe purpose of writing this article is to describe the development life cycle in a data engineering environment. I tried to describe the importance of maintaining different environments and having a deployment process. Any software or Data Engineering Development team needs a proper deployment strategy to migrate code from a…Data Engineering4 min readData Engineering4 min read
Published inGeek Culture·Feb 25, 2022Member-onlyIngesting Data from Qlik Replicate using Azure DatabricksReal-World Data Engineering Sample Project Ingesting Data from a CDC Replication tool (Qlik) that streams changes from a source database and land it on a Data Lake as JSON files into a Data Lake Analytics platform. The goal of this article is to describe a sample real-world data engineering use…Data Engineering10 min readData Engineering10 min read
