Published in Geek Culture·PinnedMy Guide to Becoming a “Market-Ready” Data Engineer With a $100 Investment in 3 Months or Less — 2021As you may know, the demand for Data Engineers has quadrupled in the last 2 years. There are more jobs than candidates in the job market, especially for US work authorized candidates. I have seen companies offering up to $1500 in referral bonuses. My goal in writing this post is…Data Engineering10 min readData Engineering10 min read
Aug 17Member-onlyTime Travel versus Slowly Changing Dimension Type 2The motivation for writing this article is to help readers realize and recognize the opportunity to utilize the Delta Lake time travel feature when faced with a problem that involves analyzing historical versions of a record (slowly changing dimension type 2). …Data Engineering5 min readData Engineering5 min read
Published in Geek Culture·Jul 12Member-onlyCracking the Data Engineering Interview — Part 2: GuideData engineering interviews can be quite frustrating especially when you get a lot of recruiter calls, and interview request that all end up with a rejection letter. In the last post, I described the typical data engineering process and offered some specific tips for each stage. …Data6 min readData6 min read
Published in Geek Culture·Jul 7Member-onlyCracking the Data Engineering Interview — Part 1: StructureThe goal of this blog post is to offer some general information about the data engineering interview process and offer some useful tips that might help a newbie land their first data engineering job. As a senior data engineer that has transitioned from a completely non-IT-related field into data engineering…Data4 min readData4 min read
Published in Geek Culture·May 3Member-onlyBeginner's Guide: Development Life Cycle in Data EngineeringThe purpose of writing this article is to describe the development life cycle in a data engineering environment. I tried to describe the importance of maintaining different environments and having a deployment process. Any software or Data Engineering Development team needs a proper deployment strategy to migrate code from a…Data Engineering4 min readData Engineering4 min read
Published in Geek Culture·Feb 25Member-onlyIngesting Data from Qlik Replicate using Azure DatabricksReal-World Data Engineering Sample Project Ingesting Data from a CDC Replication tool (Qlik) that streams changes from a source database and land it on a Data Lake as JSON files into a Data Lake Analytics platform. The goal of this article is to describe a sample real-world data engineering use…Data Engineering10 min readData Engineering10 min read
Published in Geek Culture·Feb 6Member-onlyBecoming a Data Engineer: Resume Writing TipsFor folks transitioning into the IT industry specifically Data Engineering or experienced data engineers finding it hard to attract the jobs they want, I wrote this article for you. What is the goal of a resume? In my opinion, a resume is just a ticket for an interview request either…Data4 min readData4 min read
Nov 22, 2021Member-onlyShould you be implementing the Data Lakehouse Architecture?Everything I am about to say in this article is strictly my opinion. I am a big fan of Databricks, and Delta Lake and appreciate the value they bring to modern data analytics. I am currently working with an Oil & Gas client utilizing these technological tools. …Data Engineering4 min readData Engineering4 min read
Aug 30, 2021Member-onlyData Engineering Beginner’s Guide: Modularity and CheckpointingIn this article, I will discuss a very important ETL programming concept called Checkpointing. If you are reading my blog for the first time and not familiar with ETL/ELT please review my prior post about ETL. Checkpoint is nothing new in Software engineering a checkpoint literary means “a point where…Data Engineering5 min readData Engineering5 min read
Jul 17, 2021Member-onlyDesigning a Purpose-driven Data Analytics ArchitectureMany organizations aspire to streamline their data analytics architecture. Many look to the latest technological solutions that may have the magic formula to solve all their data challenges. However, I have come across few organizations that create a purpose-driven data analytics architecture. What I mean by purpose-driven is a data…Data Engineering6 min readData Engineering6 min read
