Azure Databricks Platform Administrator

Azure Databricks Platform Administrator Location - Raritan, NJJob Summary We are seeking an experienced Azure Databricks Platform Administrator to design operate secure and optimize enterprise scale Databricks environments on Microsoft Azure The role is responsible for platform governance workspace administration identity and access management networking security monitoring automation cost optimization and production support The ideal candidate combines strong Azure infrastructure knowledge with handson Databricks administration and works effectively with data engineering analytics security cloud and application teams

Key Responsibilities Administer Azure Databricks accounts work spaces clusters cluster policies pools SQL warehouses jobs notebooks repositories libraries and access controls across development test and production environments

Manage workspace provisioning and configuration using Infrastructure as Code and automated deployment pipelines

Implement identity and access management through Microsoft Entra e signon service principals managed identities groups rolebased access control and SCIM provisioning

Administer Unity Catalog including metastores catalogs schemas tables volumes storage credentials external locations privileges and data access governance

Configure secure Azure networking including virtual network injection private endpoints Private Link network security groups route tables DNS firewalls and controlled egress

Integrate Databricks with Azure Data Lake Storage Key Vault Azure Dev Ops or Git Hub monitoring platforms and other approved enterprise services

Establish cluster standards runtime policies library controls tagging autoscaling autotermination and workload isolation

Monitor platform availability performance capacity audit logs job health and security events build dashboards and actionable sLead incident problem change release and service request management for the platform meeting agreed service levels and operational KPIsTroubleshoot complex workspace compute authentication authorization connectivity storage Spark and integration issues

Optimize platform cost through usage analysis rightsizing policy enforcement commitment utilization chargebackshowback and removal of idle resources

Plan and execute Databricks Runtime upgrades platform changes vulnerability remediation business continuity testing and disaster recovery activities

Define and maintain platform standards runbooks architecture diagrams operating procedures support documentation and knowledge articles

Partner with engineering architecture security compliance Fin Ops and vendor teams to onboard workloads and drive continuous improvement

Mentor junior administrators and provide technical leadership for platform operations and governance

Required Qualifications and Experience5 years of experience in cloud data platform infrastructure Dev Ops or production support roles including substantial handson administration of Azure Databricks Strong understanding of Azure Databricks architecture control plane and data plane concepts work spaces compute jobs SQL warehouses Delta Lake and Unity Catalog Handson experience with core Azure services such as Microsoft Entra  Data Lake Storage Gen2 Key Vault Virtual Network Private Link Azure Monitor Log Analytics and Azure Dev Ops Strong  knowledge of cloud security least privilege access secrets management encryption network isolation auditability and enterprise governance

Experience automating Azure and Databricks administration using Terraform Bicep or ARM templates Databricks Asset Bundles Azure CLI Databricks CLI REST APIs Power Shell Python or shell scripting

Experience building and supporting CICD pipelines for infrastructure and platform configuration

Working knowledge of…