Generative AI Engineer

Magpie Health Analytics is seeking an experienced AI Engineer to design, build, and operate secure, trustworthy generative AI solutions that help our healthcare clients deliver faster, more consistent access to authoritative program guidance.

Our clients are government health payers. Your primary responsibility is to design, develop, test, deploy, and maintain an AI-enabled decision-support capability that allows users to search and navigate large bodies of program documentation (statutes, regulations, rules, and guidance) using natural language. The solution will first support internal customers and will later be extended to external, public-facing users. You will work in collaboration with project managers, data engineers, full stack developers, analysts, and client stakeholders to ensure the successful implementation of the client solution.

The AI Engineer duties include:

• Design, develop, and maintain generative AI solutions that ground large language model (LLM) responses in authoritative source documentation.
• Build and maintain document ingestion pipelines, including parsing, chunking, metadata tagging, embedding generation, and vector/semantic search indexing.
• Collaborate with cross-functional teams and subject matter experts to understand user needs and translate them into requirements, prompts, and evaluation criteria.
• Implement responses with source citations and direct links to the underlying documentation to ensure traceability.
• Design and implement guardrails, including content filtering, scope restrictions, PII/PHI protections, and refusal handling for out-of-scope questions.
• Develop evaluation frameworks and test sets to measure retrieval quality, answer accuracy, groundedness, and consistency; monitor and tune performance over time.
• Deploy and operate solutions within approved AWS services and federal security controls, supporting documentation needed for security authorization and AI governance reviews.
• Implement logging, monitoring, and audit trails to support transparency, cost management, and responsible AI oversight.
• Write clean, reusable, and well-documented code, and participate in code reviews, testing, and deployment processes.
• Stay up to date with emerging AI technologies, models, and federal AI guidance.

• A master’s degree in computer science, data science, engineering, or another relevant field.

• 5+ years of experience developing software and cloud-based solutions, with demonstrated recent experience designing and deploying LLM or generative AI applications.
• Strong proficiency in Python and experience with modern LLM APIs, SDKs, orchestration frameworks, and AI/ML libraries (e.g., Hugging Face, Lang Chain, Llama Index, or equivalent).
• Hands-on experience designing and deploying generative AI solutions, including embeddings, vector databases, and hybrid/semantic search.
• Experience with AWS AI and data services such as Amazon Bedrock, Amazon Open Search, Amazon Kendra, Sage Maker, Lambda, or S3.
• Experience with prompt and context engineering, LLM evaluation, and techniques for reducing hallucinations and ensuring answers are grounded in source material.
• Designing AI applications to support multiple foundation models/providers and evaluating models based on accuracy, latency, cost, security, and use-case requirements.
• Experience with Agile software development methodologies.
• Experience with building and maintaining CI/CD pipelines, testing, and deploying infrastructure using services such as Git Hub Actions, Cloud Formation, Terraform, and/or Jenkins.
• Excellent problem-solving and communication skills, including the ability to explain AI capabilities and limitations to non-technical stakeholders.
• Ability…