Lead Data Engineer

Qualifications

• 10+ years of hands‑on data engineering experience, including technical leadership of data platform initiatives
• Strong, hands‑on expertise with the Databricks Lakehouse Platform, including Delta Lake, Delta Live Tables, Databricks Workflows, and Unity Catalog
• Proven experience designing and implementing the medallion (bronze/silver/gold) architecture for data ingestion, curation, and consumption
• Strong expertise in distributed data processing using Apache Spark (PySpark and Spark SQL) or equivalent big‑data frameworks
• Proven experience designing and building ETL/ELT pipelines for both batch and streaming data at scale
• Expert‑level proficiency in Python and SQL for data transformation, validation, and pipeline development
• Strong experience building and managing data lakes and data warehouses, including dimensional and lakehouse data modeling
• Working knowledge of Large Language Models (LLMs) and GenAI concepts — prompts, embeddings, vector databases, and Retrieval‑Augmented Generation (RAG) — and the data pipelines required to support them
• Hands‑on experience with at least one major cloud platform (AWS, Azure, or GCP) and its core data services
• Proven experience leading large‑scale data migrations from on‑premise (e.g. Hadoop) and cloud platforms to modern data architectures
• Strong understanding of distributed data processing, partitioning, and performance optimization techniques
• Experience implementing data security, governance, lineage, and access control (e.g. IAM, encryption, cataloging)
• Strong understanding of object‑oriented programming, software design patterns, and CI/CD practices
• Familiarity with Agile/Scrum delivery methodologies and experience mentoring engineers
• Excellent problem‑solving, analytical, and stakeholder‑management skills
• Strong verbal and written communication skills