Senior Data Engineer

Description

The U.S. Pharmacopeial Convention (USP) is an independent scientific organization that collaborates with the world’s leading health and science experts to develop rigorous quality standards for medicines, dietary supplements, and food ingredients. At USP, we believe that scientific excellence is driven by a commitment to fairness, integrity, and global collaboration. This belief is embedded in our core value of Passion for Quality and is demonstrated through the contributions of more than 1,300 professionals across twenty global locations, working to strengthen the supply of safe, high-quality medicines worldwide.

At USP, we value inclusive scientific collaboration and recognize that attracting diverse expertise strengthens our ability to develop trusted public health standards. We foster an organizational culture that supports equitable access to mentorship, professional development, and leadership opportunities. Our partnerships, standards, and research reflect our belief that ensuring broad participation in scientific leadership results in stronger, more impactful outcomes for global health.

USP is proud to be an equal employment opportunity employer (EEOE) and is committed to ensuring fair, merit‑based selection processes that enable the best scientific minds—regardless of background—to contribute to advancing public health solutions worldwide. We provide reasonable accommodations to individuals with disabilities and uphold policies that create an inclusive and collaborative work environment.

We are seeking a high‑energy, hands‑on Data Engineer to design, build, and maintain scalable, production‑grade data pipelines across our AWS‑based data platform. This role will own end‑to‑end pipeline delivery—from enterprise system ingestion to creating analytics‑ready datasets—while working closely with Data Architects, Analytics teams, and business stakeholders. You will play a key role in ensuring high‑quality, trusted data is accessible across the organization to support analytics and strategic decision‑making.

As part of our core data platform team, you will contribute to the continued evolution of our AWS Lakehouse architecture and help shape best practices.

As part of our mission to advance scientific rigor and public health standards, you will play a vital role in increasing global access to high‑quality medicines through public standards and related programs. USP prioritizes scientific integrity, regulatory excellence, and evidence‑based decision‑making to ensure health systems worldwide can rely on strong, tested, and globally relevant quality standards.

Additionally, USP’s People and Culture division, in partnership with the Equity Office, invests in leadership and workforce development to equip all employees with the skills to create high‑performing, inclusive teams. This includes training in equitable management practices and tools to promote engaged, collaborative, and results‑driven work environments.

Data Pipeline Development: Design, develop, and maintain scalable ELT pipelines with reusable ingestion frameworks supporting batch and event‑driven workflows across diverse sources including ERPs, APIs, vendor files, and relational databases.

AWS Cloud Data Engineering: Build and optimize AWS‑native data pipelines implementing a Medallion Lakehouse architecture with a focus on performance and cost efficiency.

Infrastructure, CI/CD & Dev Ops: Provision and manage AWS infrastructure via Terraform, maintaining CI/CD pipelines with Git Ops practices, automated testing, logging, alerting, and recovery mechanisms for production systems.

Data Modeling & Transformation: Design and implement dimensional models, Gold‑layer datasets, and scalable vendor file ingestion patterns using SQL, Python, and PySpark, with robust handling of schema drift, audit tracking, and query optimization through partitioning, clustering, and materialization strategies.

Data Quality & Governance: Implement automated data validation, anomaly detection, and lineage tracking within pipelines…