AI & Data Engineer, Data Discovery Services

Position Summary

Join the Data Discovery Services team within Enterprise Data Platforms, where we deliver and maintain the data foundation platforms that power discovery, search, and data accessibility across the Bristol Myers Squibb enterprise. We operate multiple search and discovery services used across the company and are building the next generation of AI‑powered discovery on top of them. Our work makes enterprise data findable, accessible, and actionable for teams across the organization.

At the core of this is a semantic knowledge layer—metadata, taxonomies, and relationships that describe what data means and how it connects—curated as a data inventory that helps AI work reliably across the enterprise. This is a high‑impact team where engineering, search, and applied AI come together to solve real problems at scale.

As an AI/Data Engineer, you will be a hands‑on Python developer building pipelines and integrations that make enterprise data more discoverable. Your primary focus is data engineering—pipelines, metadata enrichment, transformations, and platform integrations—and you will also contribute to search and AI‑powered retrieval as you grow into the role.

• Build and maintain Python pipelines that pull metadata from enterprise data catalogs, enrich it with taxonomy tags and ownership information, and publish it to the discovery platform.
• Tune and optimize search indexes—adjust analyzers, boost fields, and test queries—to ensure results match user needs.
• Build a semantic knowledge layer—chunk documents, generate vector embeddings, and enrich them with semantic knowledge metadata—to grow a data inventory that supports retrieval‑augmented generation (RAG) and helps AI systems and large language models find and use the right context.
• Maintain integrations that sync ontology and taxonomy changes into the discovery platform so classifications stay current.
• Investigate and resolve data pipeline issues across Databricks and AWS Glue, trace root causes through metadata enrichment flows, and add data quality checks to prevent recurrence.
• Build API endpoints and Model Context Protocol (MCP) servers that expose search and metadata capabilities to applications and AI agents.
• Design metadata pipelines that map cross‑domain dataset relationships and add them to the cross‑domain join catalog with confidence scores.
• Analyze search patterns, capture user feedback, and improve the discovery experience so the system learns and improves over time.

• Bachelor’s degree in Computer Science, Data Science, Information Science, Engineering, or a related field. Master’s degree preferred.
• Demonstrated proficiency in data engineering, software engineering, or a related technical discipline, with a track record of delivering production data pipelines.
• Proficient Python skills—this is your primary language day‑to‑day.
• Proficiency in Structured Query Language (SQL).
• Experience with Databricks and AWS Glue for data pipelines and transformations.
• Solid data engineering fundamentals: ETL/ELT patterns, data modeling, data quality, and pipeline orchestration.
• Familiarity with AWS cloud services (S3, Lambda, API Gateway, Glue).
• Experience with Open Search or Elasticsearch.
• Understanding of metadata management and data cataloging concepts.
• Effective problem‑solving skills and willingness to learn.
• Good communication skills and ability to work collaboratively in a team.

• Experience with semantic knowledge layers, RAG patterns, vector search technologies, and building AI‑ready data inventories.
• Familiarity with semantic search, embeddings, chunking strategies, relevance tuning, and semantic knowledge metadata (entity relationships, taxonomies, context enrichment).
• Exposure to AI agent patterns, MCP, or large language model orchestration frameworks.
• Experience with ontology or taxonomy technologies (such as Turtle, RDF, OWL, or SPARQL) or management platforms.
• Familiarity with graph databases or knowledge graph technologies.
• Experience with metadata enrichment, data lineage, or data quality frameworks.
• Exposure to Azure OpenAI, Google Vertex AI, or Amazon Bedrock.
• Experienc…