Manager, Data Engineering

Responsibilities

• Lead, coach, and develop a team of Data Engineers, fostering strong technical skills and ownership
• Set clear priorities, manage workload, and ensure timely delivery of data engineering initiatives
• Establish engineering standards, code quality expectations, and best practices across the team
• Partner with stakeholders to translate business needs into scalable data solutions
• Oversee the design, development, and operation of data pipelines built in Databricks
• Ensure pipelines are scalable, reliable, and aligned to medallion architecture standards (bronze, silver, gold)
• Guide implementation of ingestion frameworks using tools like Fivetran and custom ingestion patterns
• Provide technical leadership for the Databricks platform, including workspace, jobs, clusters, and performance optimization
• Collaborate with Cloud Engineering to ensure seamless integration with AWS services (e.g., S3)
• Ensure alignment with enterprise architecture, including data modeling, partitioning strategies, and storage optimization
• Optimize compute usage for performance and cost efficiency
• Establish and enforce data quality standards, including testing, validation, and monitoring frameworks
• Ensure robust observability across pipelines (monitoring, alerting, lineage visibility)
• Partner with Governance teams to support metadata management and lineage through tools like Atlan
• Enforce security, compliance, and data access standards across all data engineering assets
• Own production support processes, including incident management, root cause analysis, and prevention
• Establish proactive monitoring and health checks for pipelines and platform performance
• Drive continuous improvement in automation, CI/CD, and release management practices
• Support and lead data migration efforts from legacy on-prem systems to cloud platforms

• Bachelor’s Degree or equivalent experience
• 7+ years of experience in data engineering or data platform roles
• 3+ years of experience leading and developing technical teams
• Experience operating in modern cloud data environments (Databricks, data lakes, or lakehouse platforms)
• Proven experience delivering enterprise-scale data pipelines and platforms
• Experience supporting cloud migration or modernization initiatives
• Strong expertise in Databricks (notebooks, jobs, cluster management)
• Proficiency in Python and PySpark for distributed data processing
• Advanced SQL skills and experience working with large-scale datasets
• Experience designing and operating cloud-based data platforms (AWS preferred)
• Experience with data ingestion tools (e.g., Fivetran or similar)
• Deep understanding of data modeling and medallion architecture patterns