Senior Data Engineering Lead — Databricks & Cloud Platform

• Lead, coach, and develop a team of Data Engineers, fostering strong technical skills and ownership

• Set clear priorities, manage workload, and ensure timely delivery of data engineering initiatives

• Establish engineering standards, code quality expectations, and best practices across the team

• Partner with stakeholders to translate business needs into scalable data solutions

• Oversee the design, development, and operation of data pipelines built in Databricks

• Ensure pipelines are scalable, reliable, and aligned to medallion architecture standards (bronze, silver, gold)

• Guide implementation of ingestion frameworks using tools like Fivetran and custom ingestion patterns

• Provide technical leadership for the Databricks platform, including workspace, jobs, clusters, and performance optimization

• Collaborate with Cloud Engineering to ensure seamless integration with AWS services (e.g., S3)

• Ensure alignment with enterprise architecture, including data modeling, partitioning strategies, and storage optimization

• Optimize compute usage for performance and cost efficiency

• Establish and enforce data quality standards, including testing, validation, and monitoring frameworks

• Ensure robust observability across pipelines (monitoring, alerting, lineage visibility)

• Partner with Governance teams to support metadata management and lineage through tools like Atlan

• Enforce security, compliance, and data access standards across all data engineering assets

• Own production support processes, including incident management, root cause analysis, and prevention

• Establish proactive monitoring and health checks for pipelines and platform performance

• Drive continuous improvement in automation, CI/CD, and release management practices

• Support and lead data migration efforts from legacy on-prem systems to cloud platforms

Requirements

• Bachelor’s Degree or equivalent experience
• 7+ years of experience in data engineering or data platform roles
• 3+ years of experience leading and developing technical teams
• Experience operating in modern cloud data environments (Databricks, data lakes, or lakehouse platforms)
• Proven experience delivering enterprise-scale data pipelines and platforms
• Experience supporting cloud migration or modernization initiatives
• Strong expertise in Databricks (notebooks, jobs, cluster management)
• Proficiency in Python and PySpark for distributed data processing
• Advanced SQL skills and experience working with large-scale datasets
• Experience designing and operating cloud-based data platforms (AWS preferred)
• Experience with data ingestion tools (e.g., Fivetran or similar)
• Deep understanding of data modeling and medallion architecture patterns

🔍 ATS Optimization Keywords
Below are skills and terms extracted directly from this job posting to improve Applicant Tracking System (ATS) visibility. This unique feature helps candidates tailor their applications more effectively — a feature exclusive to Job Tailor job listings.

Hard Skills

• Python
• Py Spark
• SQL
• Databricks
• data pipelines
• data modeling
• medallion architecture
• data ingestion
• cloud migration
• CI/CD

Soft Skills

• leadership
• coaching
• communication
• prioritization
• collaboration
• problem-solving
• technical guidance
• team development
• incident management
• continuous improvement

Certifications & Qualifications

• Bachelor’s Degree