Published in Geek Culture·PinnedMy Guide to Becoming a “Market-Ready” Data Engineer With a $100 Investment in 3 Months or Less — 2021As you may know, the demand for Data Engineers has quadrupled in the last 2 years. There are more jobs than candidates in the job market, especially for US work authorized candidates. I have seen companies offering up to $1500 in referral bonuses. My goal in writing this post is…Data Engineering10 min read
May 3Beginner's Guide: Development Life Cycle in Data EngineeringThe purpose of writing this article is to describe the development life cycle in a data engineering environment. I tried to describe the importance of maintaining different environments and having a deployment process. Any software or Data Engineering Development team needs a proper deployment strategy to migrate code from a…Data Engineering4 min read
Published in Geek Culture·Feb 25Ingesting Data from Qlik Replicate using Azure DatabricksReal-World Data Engineering Sample Project Ingesting Data from a CDC Replication tool (Qlik) that streams changes from a source database and land it on a Data Lake as JSON files into a Data Lake Analytics platform. The goal of this article is to describe a sample real-world data engineering use…Data Engineering10 min read
Published in Geek Culture·Feb 6Becoming a Data Engineer: Resume Writing TipsFor folks transitioning into the IT industry specifically Data Engineering or experienced data engineers finding it hard to attract the jobs they want, I wrote this article for you. What is the goal of a resume? In my opinion, a resume is just a ticket for an interview request either…Data4 min read
Nov 22, 2021Should you be implementing the Data Lakehouse Architecture?Everything I am about to say in this article is strictly my opinion. I am a big fan of Databricks, and Delta Lake and appreciate the value they bring to modern data analytics. I am currently working with an Oil & Gas client utilizing these technological tools. …Data Engineering4 min read
Aug 30, 2021Data Engineering Beginner’s Guide: Modularity and CheckpointingIn this article, I will discuss a very important ETL programming concept called Checkpointing. If you are reading my blog for the first time and not familiar with ETL/ELT please review my prior post about ETL. Checkpoint is nothing new in Software engineering a checkpoint literary means “a point where…Data Engineering5 min read
Jul 17, 2021Designing a Purpose-driven Data Analytics ArchitectureMany organizations aspire to streamline their data analytics architecture. Many look to the latest technological solutions that may have the magic formula to solve all their data challenges. However, I have come across few organizations that create a purpose-driven data analytics architecture. What I mean by purpose-driven is a data…Data Engineering6 min read
Published in Geek Culture·Jun 27, 2021Cloud Data Lake vs Cloud Data Warehouse -Pros and ConsThis article is targeted at organizations that are recently migrating their data analytics to the Cloud or want to improve their existing architecture. The goal of this article is to present some points around the use, pros, and cons of Data Lake and Data warehouse solutions in the cloud. If…Data6 min read
Published in Geek Culture·Jun 1, 2021Beginner’s Guide: Extract Transform Load (ETL) Playbook- Incremental load design pattern Part 3The intended audience of this article are IT enthusiasts and beginner-level data engineers interested in understanding some data engineering principles In the previous post (read here), I described in detail an incremental load design pattern and solutions. In this article, I will explore the methods of inserting and updating data…Data Engineering6 min read
May 10, 2021Beginner’s Guide: Extract Transform Load (ETL) Playbook- Incremental load design pattern Part2The intended audience of this article are IT enthusiasts and beginner-level data engineers interested in understanding some data engineering principles In the previous post (read here), I described the various incremental data extraction design considerations, and requirements. …Data Science4 min read
