Lead Data Engineer

Emory University is a leading research university that fosters excellence and attracts world-class talent to innovate today and prepare leaders for the future. We welcome candidates who can contribute to the excellence of our academic community.

The Lead Data Engineer provides technical leadership and hands-on engineering expertise for the development and operation of Emory University’s Unified Data Platform (UDP), built on Microsoft Fabric. The role leads to the integration of data from HR, Finance, Student, Research, and other enterprise applications to support analytics, reporting, research, and institutional decision-making.

Working closely with architects, researchers, business stakeholders, and cross-functional teams, the lead translates requirements into reliable, scalable, and governed data solutions. This position guides engineering delivery, mentors team members, establishes development standards, and resolves complex technical issues while contributing directly to pipeline development and production support. Microsoft Fabric experience is preferred. Candidates with strong experience in Azure Synapse or Databricks with PySpark, and the ability to apply that experience to Microsoft Fabric, will also be considered.

The role requires sound judgment in data quality, security, governance, and the handling of sensitive university and research data.

KEY RESPONSIBILITIES:

• Lead the technical delivery of source system onboarding and data integration initiatives, from source analysis and design through development, validation, deployment, and production support.
• Partner with architects to implement platform designs, reusable ingestion frameworks, and engineering standards aligned with the UDP architecture and institutional priorities.
• Design, develop, and optimize ETL and ELT pipelines using SQL, Python, PySpark, and Microsoft Fabric or comparable cloud technologies.
• Analyze source schemas, business rules, and dependencies across enterprise applications; define source-to-target mappings, incremental load strategies, and reconciliation requirements.
• Guide implementation of bronze, silver, and gold data layers, including data transformation, dimensional modeling, and curated datasets for reporting and analytics.
• Establish and review standards for coding, testing, documentation, version control, and deployment; conduct code reviews and mentor data engineers.
• Coordinate engineering tasks and technical dependencies with Business Analysts, Project Managers, Data Analysts, Architects, and source system owners; communicate delivery progress, risks, and resource needs.
• Implement data quality checks, reconciliation controls, monitoring, alerting, error handling, and recovery procedures to support reliable production processing.
• Troubleshoot complex pipeline and data issues; tune SQL, Spark workloads, and orchestration to improve performance, scalability, and efficient use of platform capacity.
• Apply institutional data governance and security standards, including access controls, metadata, lineage, and appropriate handling of sensitive data; support HIPAA requirements where applicable.
• Work with researchers and business stakeholders to translate requirements into technical specifications and explain design choices and data limitations clearly.
• Create and maintain technical documentation, operational runbooks, source mappings, and support procedures; coordinate production deployments and knowledge transfer.
• Support existing UDP and Enterprise Business Intelligence (EBI) integrations and services, balancing operational needs with project delivery priorities.
• Evaluate emerging technologies and develop focused proofs of concept that address identified business or…