As you may know, the demand for Data Engineers has quadrupled in the last 2 years. There are more jobs than candidates in the job market, especially for US work authorized candidates. I have seen companies offering up to $1500 in referral bonuses.
My goal in writing this post is to give a self-motivated beginner a “simpler” learning path to becoming a data engineer. I have outlined the most important foundational skills needed to break into a Data Engineer role (junior role at least). The data engineer journey or landscape is truly overwhelming and I can say confidently there is…
The intended audience of this article are IT enthusiasts and beginner-level data engineers interested in understanding some data engineering principles
In the previous post (read here), I described in detail an incremental load design pattern and solutions. In this article, I will explore the methods of inserting and updating data in the target table and discuss near real-time incremental data extraction processes using Change Data Capture (CDC) and streaming changed data with Kafka.
Methods of Applying Source Table Transactions
- INSERTS
When a brand new record is created in the source table and is picked up by the incremental extraction process…
The intended audience of this article are IT enthusiasts and beginner-level data engineers interested in understanding some data engineering principles
In the previous post (read here), I described the various incremental data extraction design considerations, and requirements. In this article, I will continue from where I left off to explain the sample solution approach.
Example of Incremental Source Database Table Extraction to a Staging Table using ETL tool
One of the best practices when designing an incremental extraction process from a source system that involves multiple tables is to use an ETL control table.
ETL Control Table
This is the…
The intended audience of this article are IT enthusiasts and beginner-level data engineers interested in understanding some data engineering principles
In the previous post (read here), I explored the rationale behind ETL and talked about how we can achieve full data extraction from a source system to a target system. In this post, I would focus our attention on incremental data extraction and some common design patterns.
We know that as we run analytics on our downstream systems, it needs to be on updated data. However, there are different ways of updating our downstream analytical system. One of them is…
Purpose: The goal of this article is to give an introductory guide on some basics of ETL as it relates to Data Engineering.
One of the most important and often overlooked core facets of data engineering is the creation of ETL pipelines. With the popularity of AI and ML projects and the concentration of demand for data scientists. It is easy to deem ETL as an old-fashioned approach to modern data analytics solutions. I have seen so many training programs overlook or give little attention to this subject area when teaching students about data engineering. ETL or ELT is actually more fundamental and necessary before any AI/ML or Data analytics project can be kicked off. …
The motivation for writing this is to explain the major difference between SMP and MPP platforms. I will also explain their appropriate use cases, pros, and cons.
Symmetric Multi-Processor (SMP) Architecture
“Symmetric multiprocessing (SMP) involves a multiprocessor computer hardware and software architecture where two or more identical processors are connected to a single, shared main memory, have full access to all input and output devices, and are controlled by a single operating system instance that treats all processors equally, reserving none for special purposes” reference
Reference: http://shahfaisalmuhammed.blogspot.com/2015/12/SMP-versus-MPP-architecture-.html
Let’s start with some history on analytical databases
Relational databases (ex SQL Server…
The goal of this article to illustrate step-by-step how to set up an Azure Application Gateway for your web services using the Azure Portal.
Reference: https://docs.microsoft.com/en-us/azure/application-gateway/create-url-route-portal
Before we begin with this tutorial, I would like to level set us with some basic understanding of the various components we will be configuring. I think it is crucial to understand what we are doing before doing it. Here are some basic questions one should know
- What is an application gateway?
- What is a reverse proxy?
3. What are the real-world use cases for Application Gateway?
4. What is the difference between HTTP…
This article is targeted towards IT professionals with less than 2 years of experience. After a few years in the industry here are some tips I would give folks starting on this journey.
It is better to admit that you don’t know and educate yourself than to assume you know and mislead yourself or someone. Always be open to learning something new, doing your research, and learning on your own.
Personally, this year has been humbling as I have been involved in various projects that are not strictly in my data engineering domain. I am not ashamed to say that…
My goal in writing this is to help someone that is yet to start, started but frustrated, or close to achieving their goal in their data analytics journey, gain inspiration and hope from my story. I hope they get encouraged by my countless failures then eventual moments of success.
I graduated from City College of New York with a bachelor’s in electrical engineering class of 2014. In my major, I took only 2 programming classes, but they were my favorite classes in my undergraduate degree. The first was Intro to programming in C/C++, the other was an advanced class in…
AfroInfoTech
I am passionate about empowering and encouraging people of color in the data analytics career path
