Lead AI Engineer, Agentic Data Lineage

Lead AI Engineer, Agentic Data Lineage
What You Will Do

Define and drive the technical strategy for a Firmwide, column-level data lineage platform built on AI reasoning agents.

Architect parsers that build Abstract Syntax Trees (ASTs) from SQL across multiple dialects, stored procedures, Python, Java, Scala, and Spark jobs, then resolve column-level transformations from those trees.

Design ingestion of no-code signals from ETL and transformation tools, workflow orchestrators, data catalogs, and BI platforms, normalizing them into a common lineage model.

Build a multi-agent system in which specialized deep agents parse, reason, disambiguate, and reconcile conflicting signals, then assemble a coherent end-to-end lineage graph.

Establish how agents handle ambiguity (dynamic SQL, runtime configuration, hand-written transformations) by reasoning over context rather than failing silently or guessing.

Model and persist lineage as a queryable knowledge graph supporting impact analysis, root-cause tracing, and regulatory evidence generation.

Build confidence scoring and validation so every lineage edge carries a provenance signal and a measure of certainty, with low‑confidence inferences surfaced for human review.

Partner with the Firmwide Data Office, Risk, Compliance, and the divisional technology teams to align the platform with the Firm's data governance obligations, including BCBS 239, CCAR, GDPR, and SOX.

Lead, mentor, and grow a team of AI and data engineers, setting engineering standards, reviewing designs, and owning delivery.

Operate within Morgan Stanley's controls for AI safety, data confidentiality, and model governance.

Required Qualifications

8+ years of software and data engineering experience, including a track record of leading complex platform builds end to end.

Strong background in program analysis or compiler techniques: parsing, ASTs, intermediate representations, and static analysis. Hands‑on experience with tools such as ANTLR, tree‑sitter, sqlglot, or language‑native AST libraries.

Practical experience building LLM‑powered agents or multi‑agent systems, including reasoning loops, tool use, retrieval, and orchestration frameworks (Lang Graph, Llama Index, Auto Gen, or equivalent in‑house systems).

Deep data engineering expertise: SQL fluency across dialects, plus experience with Spark, distributed pipelines, and modern transformation tooling such as dbt.

Experience modeling and querying graph data, with a graph database such as Neo4j, Amazon Neptune, or Janus Graph.

Strong software engineering fundamentals: testing, CI/CD, and production‑grade systems at scale.

Demonstrated technical leadership, including mentoring engineers and influencing architecture across teams.

Preferred Qualifications

Experience in financial services, capital markets, or another heavily regulated data environment.

Familiarity with data lineage and metadata standards and tools such as Open Lineage, Marquez, Data Hub, Apache Atlas, Egeria, Collibra, or Alation.

Working knowledge of BCBS 239 data aggregation and risk reporting principles and how they translate into engineering requirements.

Experience harvesting metadata from enterprise ETL tools (Informatica, Talend) and BI platforms (Tableau, Power BI).

Background in knowledge representation, semantic reasoning, or evaluation of LLM output quality and hallucination control.

Tech Stack
Python, SQL, Spark, and Scala or Java for parsing targets. LLM and agent frameworks for reasoning. Graph databases for lineage storage. Workflow orchestration for pipeline execution. Cloud and on‑premise deployment with containerized services. Open Lineage‑aligned metadata models.

Momento USA is an Equal Opportunity/Affirmative Action Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, pregnancy, sexual orientation, gender identity, national origin, age, protected veteran status, or disability status.

#J-18808-Ljbffr