Published in Geek Culture·PinnedMy Guide to Becoming a “Market-Ready” Data Engineer With a $100 Investment in 3 Months or Less — 2021As you may know, the demand for Data Engineers has quadrupled in the last 2 years. There are more jobs than candidates in the job market, especially for US work authorized candidates. I have seen companies offering up to $1500 in referral bonuses. My goal in writing this post is…Data Engineering10 min readData Engineering10 min read
Published in Geek Culture·Jan 18Member-onlyDesigning Incremental data extraction for a source data that is a SQL queryThe goal of this article is to describe how we can design and implement an incremental ETL pipeline that extracts data from a source query (instead of the common case of a table) and load it to a Target table. Often times, most people are knowledgeable in designing incremental data…Data Engineering5 min readData Engineering5 min read
Aug 17, 2022Member-onlyTime Travel versus Slowly Changing Dimension Type 2The motivation for writing this article is to help readers realize and recognize the opportunity to utilize the Delta Lake time travel feature when faced with a problem that involves analyzing historical versions of a record (slowly changing dimension type 2). …Data Engineering5 min readData Engineering5 min read
Published in Geek Culture·Jul 12, 2022Member-onlyCracking the Data Engineering Interview — Part 2: GuideData engineering interviews can be quite frustrating especially when you get a lot of recruiter calls, and interview request that all end up with a rejection letter. In the last post, I described the typical data engineering process and offered some specific tips for each stage. …Data6 min readData6 min read
Published in Geek Culture·Jul 7, 2022Member-onlyCracking the Data Engineering Interview — Part 1: StructureThe goal of this blog post is to offer some general information about the data engineering interview process and offer some useful tips that might help a newbie land their first data engineering job. As a senior data engineer that has transitioned from a completely non-IT-related field into data engineering…Data4 min readData4 min read
Published in Geek Culture·May 3, 2022Member-onlyBeginner's Guide: Development Life Cycle in Data EngineeringThe purpose of writing this article is to describe the development life cycle in a data engineering environment. I tried to describe the importance of maintaining different environments and having a deployment process. Any software or Data Engineering Development team needs a proper deployment strategy to migrate code from a…Data Engineering4 min readData Engineering4 min read
Published in Geek Culture·Feb 25, 2022Member-onlyIngesting Data from Qlik Replicate using Azure DatabricksReal-World Data Engineering Sample Project Ingesting Data from a CDC Replication tool (Qlik) that streams changes from a source database and land it on a Data Lake as JSON files into a Data Lake Analytics platform. The goal of this article is to describe a sample real-world data engineering use…Data Engineering10 min readData Engineering10 min read
Published in Geek Culture·Feb 6, 2022Member-onlyBecoming a Data Engineer: Resume Writing TipsFor folks transitioning into the IT industry specifically Data Engineering or experienced data engineers finding it hard to attract the jobs they want, I wrote this article for you. What is the goal of a resume? In my opinion, a resume is just a ticket for an interview request either…Data4 min readData4 min read
Nov 22, 2021Member-onlyShould you be implementing the Data Lakehouse Architecture?Everything I am about to say in this article is strictly my opinion. I am a big fan of Databricks, and Delta Lake and appreciate the value they bring to modern data analytics. I am currently working with an Oil & Gas client utilizing these technological tools. …Data Engineering4 min readData Engineering4 min read
Aug 30, 2021Member-onlyData Engineering Beginner’s Guide: Modularity and CheckpointingIn this article, I will discuss a very important ETL programming concept called Checkpointing. If you are reading my blog for the first time and not familiar with ETL/ELT please review my prior post about ETL. Checkpoint is nothing new in Software engineering a checkpoint literary means “a point where…Data Engineering5 min readData Engineering5 min read
