Senior Databricks Engineer

We are seeking a Senior Databricks Engineer with deep hands‑on experience designing and implementing large‑scale data solutions on Azure Databricks
. The ideal candidate has real‑world experience building and troubleshooting production‑grade data pipelines, optimizing Spark workloads, managing Delta Lake architecture, and implementing Dev Ops best practices using IaC and CI/CD automation.

• Design, develop, and maintain data pipelines and ETL solutions in Azure Databricks using PySpark and Delta Lake.
• Implement data integration frameworks and API‑based ingestion using tools like Apigee or Kong
  .
• Analyze, design, and deliver enterprise data architecture solutions focusing on scalability, performance, and governance.
• Implement automation tools and CI/CD pipelines using Jenkins, Ansible, or Terraform.
• Troubleshoot production failures and performance bottlenecks—fix partitioning, caching, shuffle, cluster sizing, and Z‑ordering issues.
• Manage Unity Catalog
  , enforce data security (row/column‑level access), and maintain data lineage.
• Administer Databricks clusters, jobs, and SQL warehouses
  , optimizing costs through auto‑stop, job clusters, and Photon usage.
• Collaborate with cross‑functional teams to drive data strategy and standards across domains.
• Create and maintain detailed architectural diagrams
  , interface specs, and data flow documentation.
• Mentor junior engineers on Databricks, Spark optimization, and Azure data best practices.

• 5+ years of experience as a Data Engineer with strong hands‑on experience in Azure Databricks and Py Spark .
• Solid understanding of Delta Lake
  , Z‑ordering
  , partitioning
  , OPTIMIZE
  , and ACID transactions
  .

Mid‑Senior level

Contract

Information Technology

IT Services and IT Consulting

Cincinnati, OH

$91,520.00 - $