Lead Principal Data Engineer

Lead / Principal Data Engineer

Peter Millar is an American lifestyle brand seeking a technical owner to design, build, and operate our modern data platform built on Microsoft Fabric and One Lake.

• Platform Architecture & Ownership – Own the end‑to‑end architecture of the Microsoft Fabric environment, including Lakehouse and Warehouse design, the bronze/silver/gold medallion pattern, and Direct Lake semantic datasets.
• One Lake Design & Governance – Design and govern One Lake as the organization‑wide data lake, managing domains, shortcuts, data mirroring, and Delta Lake/Parquet standards.
• Data Ingestion & Transformation – Own data ingestion and transformation at scale using Fabric Data Factory pipelines, Dataflows Gen2, and Spark notebooks.
• Engineering Standards – Define and enforce Git‑based source control, CI/CD and deployment pipelines, environment promotion, and code review.
• Reliability, Performance & Cost – Establish monitoring, alerting, and observability across pipelines; ensure SLAs for data freshness and availability. Optimize Fabric capacity units and storage layout for performance and cost. Implement data quality, lineage, and automated testing frameworks.
• AI‑Ready Foundation & Generative AI Enablement – Structure well‑modeled data in One Lake to support generative and agentic AI use cases. Partner with the AI Engineering Manager to integrate Azure AI services and Microsoft Foundry into production workflows, including retrieval (RAG) pipelines over One Lake. Implement security and governance controls, row/column‑level security, and mandatory PII handling for AI at scale.
• Leadership & Vendor Management – Direct and set the technical agenda for outsourced engineering partners, shifting them from platform operation to project‑based extensions. Serve as a technical hiring manager and interviewer for new engineering hires and mentor junior analytics and data engineers. Capture and retain institutional knowledge of architecture and design patterns in‑house.

• Deep, hands‑on experience with Microsoft Fabric – Lakehouse, Warehouse, Data Factory pipelines, Dataflows Gen2, Spark notebooks, and Direct Lake.
• Demonstrated experience designing and governing One Lake, including domains, shortcuts, mirroring, and Delta Lake/Parquet standards.
• Experience integrating Azure AI services / Azure OpenAI into data and retrieval pipelines.
• Working knowledge of Microsoft Foundry (Azure AI Foundry) model catalog and RAG/agent grounding for GenAI workloads.
• Experience with Azure data platform components such as Synapse, Azure Data Lake Storage, Azure SQL, Service Bus, and Spark in production.
• Strong CI/CD, Git‑based source control, and infrastructure‑as‑code practices.

• 8+ years in data engineering / data architecture, including 2+ years in a lead or principal capacity.
• Expertise in SQL and Python; strong PySpark experience.
• Proven track record building and operating a Microsoft Fabric / One Lake platform in production.
• Experience managing or directing outsourced/consulting engineering teams.
• Exposure to Microsoft Purview, data governance, and security/compliance frameworks.
• Bachelor’s degree in Computer Science, Engineering, or related field;
  Master’s degree or relevant Microsoft certifications (e.g., Fabric Analytics Engineer, Azure Data Engineer Associate) a plus.

Peter Millar & G/FORE are equal‑opportunity employers. In accordance with anti‑discrimination law, the purpose of this policy is to effectuate principles and mandates that prohibit discrimination and harassment by race, color, religion, gender, age, national origin, genetic information, marital status, disability status, protected veteran status, sexual orientation, or any other characteristic protected by law.