Lead Data Engineer; Databricks

Location: [Remote]

We are seeking an experienced Lead Data Engineer (Databricks) with a strong background in IT consulting and professional services to lead end-to-end cloud data enterprise initiatives. In this role, you will act as a trusted technical advisor to client stakeholders while leading the architecture, design, and execution of scalable Lakehouse platforms on AWS using Databricks.

The ideal candidate possesses exceptional client engagement abilities, deep hands-on expertise with the Databricks Medallion Architecture
, Unity Catalog
, and Delta Live Tables (DLT), along with a proven track record of migrating legacy data warehouses to modern AWS data ecosystems.

• Lakehouse Architecture: Architect, design, and implement scalable, enterprise-grade data platforms on AWS using the Databricks Lakehouse architecture.
• Medallion Architecture: Establish robust data architecture patterns across Bronze, Silver, and Gold layers for both real-time streaming and batch processing pipelines.
• Data Governance & Security: Implement fine-grained access control, governance, data lineage, and audit trails using Databricks Unity Catalog, AWS IAM, and KMS.
• Performance & Cost Optimization: Optimize cluster configurations, auto-scaling parameters, and Delta Lake performance techniques (Z-Ordering, Liquid Clustering, compaction).
• Ecosystem Integration: Seamlessly integrate AWS native services (S3, AWS Glue, Lambda, Event Bridge, Redshift, Athena, Secrets Manager) with Databricks.

• Stakeholder Management: Lead discovery workshops, technical assessments, and architecture reviews with C-level executives, VPs of Engineering/Data, and Enterprise Architects.
• Technical Roadmaps: Translate complex business requirements into solution blueprints, high-level architecture diagrams, technical roadmaps, and Statements of Work (SOWs).
• Legacy Modernization: Drive client migration strategies from legacy data warehouses (e.g., Teradata, Netezza, Snowflake, On-Prem Hadoop) to AWS Databricks.
• Best Practices Guidance: Educate clients on cloud data best practices, Fin Ops/cost optimization strategies, and modern Lakehouse paradigms.
• Technical Mentorship: Guide, direct, and mentor cross-functional teams of Data Engineers, Analytics Engineers, and Dev Ops Engineers.
• Infrastructure as Code (IaC): Enforce IaC best practices using Terraform or Cloud Formation for automated enterprise workspace and cluster deployments.
• Dev Ops & Automation: Build and maintain CI/CD pipelines for Databricks notebooks, Delta Live Tables (DLT), and Spark jobs across multi-environment setups (Dev, Test, Prod).

• Education: Bachelor’s or Master’s degree in Computer Science, Data Engineering, Information Technology, or a related field.
• 8+ years in Data Engineering with at least 3+ years in a Lead/Architect capacity.
• Deep experience with Databricks
  , PySpark/Scala, Delta Lake, Unity Catalog, and Delta Live Tables (DLT).
• Hands-on experience with AWS Cloud Data Ecosystem (S3, Glue, Lambda, EMR, Redshift, IAM).
• Proficiency in Python
  , SQL
  , and Spark SQL performance tuning.
• Strong experience with Infrastructure-as-Code (
  Terraform
  ) and CI/CD tools (Git Hub Actions, Azure Dev Ops, Jenkins).
• Consulting &
  
  Soft Skills:
  
  
• Exceptional verbal and written communication skills with the ability to articulate complex technical ideas to executive audiences.
• Strong problem-solving skills and client-facing advisory experience.