Agentic AI Data Engineer - CMC Data Integration

Overview

We are seeking an AI Data Engineer to build the data ingestion infrastructure and a unified data model that underpins the modernized CMC Data Backbone. This hands‑on engineering role will write production‑quality pipelines, define CMC data schemas, and work closely with scientists and digital architects to ensure data from internal LIMS/ELN systems and external CDMO partners flows reliably into a single data backbone.

• Implement individual agent components (e.g., document extraction, schema mapping, validation) within the established orchestration framework (Lang Graph, Llama Index, or equivalent).
• Write tool‑calling logic, handle failure modes, and ensure each component is testable and observable with instrumented logging of inputs, outputs, and intermediate decisions.
• Iterate on agent behavior based on real data performance; collaborate with senior engineers to identify and resolve failure patterns.
• Participate in validation and qualification activities for AI‑assisted workflows, producing documentation that demonstrates computational tools reflect scientific intent.
• Build review queues and flagging logic that surfaces low‑confidence or out‑of‑spec extraction findings to scientific reviewers for approval before data is loaded.
• Implement routing logic that captures reviewer decisions, logs outcomes with full audit trail, and reintegrates approved data into the pipeline in compliance with 21
  
  CFRPart
  11 electronic records requirements.
• Tune flagging thresholds based on feedback from scientific owners; maintain and improve HITL logic as new data sources are onboarded.
• Design and build AI‑assisted ingestion pipelines that extract and structure data from unstructured CDMO/CRO sources such as PDFs, Excel files, and vendor portal exports.
• Implement validation, reconciliation, and exception‑handling logic to ensure data completeness and integrity before loading.
• Build monitoring and alerting for pipeline health, data quality, and ingestion failures.
• Develop a data quality framework with automated checks, rejection handling, and audit trail logging.
• Create reusable pipeline templates and schema documentation that reduce onboarding time for new CDMO partners.

• MS or PhD in Computer Science, Computer Engineering, Data Engineering, or related field with 1–2years of relevant experience; or BS in Computer Science or Computer Engineering with 3–5years of hands‑on data engineering experience.
• Proficiency in Python and SQL; ability to write, review, and own production‑quality code.
• Experience building ETL/ELT pipelines from unstructured or semi‑structured sources such as PDFs, Excel, JSON, XML.
• Hands‑on experience building LLM‑powered applications: retrieval‑augmented generation, tool‑calling, multi‑step orchestration, or equivalent agentic patterns.
• Experience with cloud data platforms (Azure Data Factory, Azure Databricks, Azure Fabric; or AWS S3, Glue, Lambda, Redshift).
• Solid understanding of relational data modeling, schema design, and data normalization principles.
• Familiarity with data orchestration tools (Airflow, Azure Data Factory, Prefect, or similar).

• Knowledge of 21
  
  CFRPart
  11, ALCOA+, and GxP data integrity principles.
• Experience integrating data from LIMS, ELN, SDMS, or CDS systems such as Benchling, Lab Vantage, OpenLABS.
• Familiarity with pharmaceutical CMC data types: analytical results, batch records, stability studies, specifications.
• Experience with data mesh architecture or data product ownership models.
• Knowledge of MLOps practices and preparing data for AI/ML model training in regulated environments.
• Exposure to regulatory submission data formats: eCTD, CTD, CDISC SEND/SDTM.
• Experience with CI/CD pipelines (Git Hub Actions, Azure Dev Ops) applied to data engineering workloads.

• Annual salary range: $65,250 – $169,400, based on education, experience, skill, and location.
• Bonus eligibility for full‑time employees, depending on company and individual performance.
• Comprehensive benefits: 401(k) sponsorship, pension, vacation, medical, dental, vision, prescription drug coverage, flexible benefit plans, life and disability insurance, leave of absence, well‑being programs, employee assistance, and employee clubs.

Lilly is a proud EEO Employer and does not discriminate on the basis of age, race, color, religion, gender identity, sex, gender expression, sexual orientation, genetic information, ancestry, national origin, protected veteran status, disability, or any other legally protected status. Lilly also offers accommodations for applicants with disabilities; please visit  for assistance.