Databricks Automation Engineer

We are looking for an experienced Databricks Automation Engineer to design, implement, and scale automated testing frameworks for our data lakehouse platform. This role blends data engineering expertise with test automation to ensure high-quality, reliable, and performant data pipelines.

• Architect and maintain automated test frameworks for ETL/ELT, Databricks/Spark, APIs, and UIs across multiple teams.
• Design data validation, quality checks, and anomaly detection strategies for large-scale telemetry and batch/streaming pipelines.
• Integrate testing into CI/CD pipelines (Azure Dev Ops / Git Hub Actions) for pre-deploy gating, contract testing, and regression protection.
• Drive performance and scalability testing for Spark jobs and microservices; identify and remediate bottlenecks.
• Lead synthetic test data generation, mocking/stubbing, and secure environment-aware test harnesses.
• Implement test observability, metrics dashboards, and automated verification SLAs.
• Own test documentation, runbooks, failure triage, and production handovers.
• Mentor and grow a high-performing QA/test automation team; promote standards and best practices.

• 8+ years in software engineering/testing, with QA ownership across the SDLC.
• 3+ years leading or mentoring engineering/QA teams.
• Strong expertise in Databricks/Spark/Delta Lake, data pipelines, and lakehouse patterns (CDC, schema evolution, delta tables, time-travel, lineage).
• Hands‑on experience building scalable automated test frameworks integrated with CI/CD.
• Proficient in Python, Java, or Scala for test automation; experience with big‑data platforms at scale.