How to run Azure Databricks workflow Job as service principal (with video)

As Databricks evolves and continues to add more features to its platform, administration becomes increasingly crucial and daunting. There’s so much to learn and manage. In this blog, I outline the steps to set up the service principal and run as a service principal.

Some basics to know in Databricks workflows to ensure the job keeps running without permission issues: the “Run As” persona setting is more crucial compared to the Workflow job “Creator.”

Why should you make this change? Using service principals avoids the problem of job failure if the creator of the job leaves the organization.

Steps to changing your “Run As” to a service principal

Step 1: Create a Service Principal in Azure.

Step 2: Grant access to this service principal in Azure Databricks workspace by assigning it a contributor role. This step ensures your service principal can access the workspace and its compute resources.

Step 3: Grant access to this service principal in the Databricks Account console (user management) by adding it there. This step ensures your service principal is registered for that Databricks account.