Lead Clinical Data Engineer

Working at Yale means contributing to a better tomorrow. Whether you are a current resident of our New Haven-based community, eligible for opportunities through the New Haven Hiring Initiative, or a newcomer, interested in exploring all that Yale has to offer, your talents and contributions are welcome. Discover your opportunities at Yale!

Overview The Lead Clinical Data Engineer is a senior technical specialist responsible for leading the ongoing design and development of the Aligned Data Warehouse, a centralized data repository for biomedical research for Yale School of Medicine in Yale New Haven Health System. In this position, the Lead Clinical Data Engineer will work with manager and other members of the data warehouse team to expand the functionality and integrate new data sources into the ADW and other research databases.

The ADW is built on the Microsoft SQL Server technology stack, with data extracted from research and clinical systems, especially the Epic’s data warehouse, and transformed into OHDSI’s Observational Medical Outcomes Partnership (OMOP) Common Data Model and other data marts. The ADW is integrated with Linux and Windows-based systems for natural language processing, Large Language Models and other GPU-based software.

This work supports major national and regional research projects and data-sharing initiatives.

Responsibilities include:

Design, build, test, maintain, and control data pipelines and ETL jobs for integrating data into the ADW and other research databases. Such data pipelines are implemented in T-SQL stored procedures, SSIS, and SQL Agent jobs, Microsoft Fabric and make extensive use of metadata-driven dynamic SQL. Contribute to Dev Ops and documentation, developing standards and procedures for database and data pipeline design, operational management, and ongoing maintenance.

Design, build, test, maintain, and control other processes for data management, including management and loading of flat files in a variety of formats (e.g., csv, tsv, pipe-delimited, XML, JSON). Facilitate design sessions, code walkthroughs, peer reviews and produce technical documentation, SOPs and KBAs. Performance tune database objects, stored procedures, ETL jobs, and related scripts to optimize both end-user customer queries and data pipeline ETL processing.

Monitor scheduled ETL jobs and other processes to ensure expected functioning and uptime of data pipelines. Lead and/or coordinate the troubleshooting and remediation of all ETL job failures in a timely manner. On-call support is sometimes required. Deliver world-class customer service in all interactions with customers, stakeholders, and other teams. Maintain a customer-focused approach to provide solutions that are science/research-driven.

Maintain patient privacy and the integrity and security of healthcare data in all databases and systems, including compliance with all applicable laws, regulations, and institutional policies related to its Institutional Review Board (IRB), patient privacy, and IT cybersecurity. These laws include, but are not limited to, the “Common Rule” (45 CFR Part 46), HIPAA (45 CFR Part 164), and 42 CFR Part
2. Independently investigate and stay abreast of new and emerging technologies that Scientific Computing & Data could leverage to provide new capabilities, boosting efficiencies or quality, and/or lower operating costs

Lead projects and data stewardship strategies Work closely with DBA team and vendors to optimize data pipelines Assist with the design, build, test, and performance-tuning of reports and dashboards in SSRS, Microsoft Power BI, or similar business intelligence tools. Performs related duties as assigned or requested. Required

Skills and Abilities

1.Extensive experience…