AI Engineer

AI Engineer at Vytalize Health

As an AI Engineer at Vytalize Health, you will design, build, and maintain agentic systems and LLM-powered applications that automate complex healthcare workflows and accelerate our ability to deliver data-driven clinical solutions. Working at the intersection of applied AI and healthcare, you will build agents that orchestrate data retrieval, model inference, clinical logic, and tool use to solve problems that traditionally require manual effort or specialized expertise.

You will work cross-functionally with data engineering, platform, product, and clinical teams to identify high-impact opportunities for AI automation—from data source onboarding to clinical decision support to evidence synthesis. Your focus will be on building production-grade agentic systems with rigorous validation, clear confidence scoring, and human-in-the-loop oversight to ensure reliability in a regulated healthcare environment. You will establish patterns, best practices, and tooling that allow the organization to scale AI-driven automation across multiple domains.

You will measure agent performance and impact—tracking accuracy, hallucination rates, and real-world clinical outcomes. You will be part of a growing AI team that values both cutting-edge AI capabilities and deep healthcare domain understanding.

Primary Responsibilities

• Design, build, and maintain agentic systems and LLM-powered applications that automate healthcare workflows, data pipelines, and clinical decision support — from conception through production deployment
• Build and orchestrate agents using LLM APIs (OpenAI, Anthropic, etc.) and agentic frameworks (Lang Chain, Lang Graph, CrewAI, or custom orchestration) to solve complex, multi-step healthcare problems
• Develop prompt libraries, agent instructions, and reusable "skills" that improve agent accuracy, consistency, and reliability across different use cases and data domains
• Build validation and confidence-scoring layers that flag low-confidence agent decisions for human review before production deployment; establish guardrails and review workflows for agent-authored code and outputs
• Own end-to-end delivery of AI-automated systems — from problem scoping and requirements gathering through agent development, testing, and validated production deployment
• Implement rigorous evaluation and QA frameworks for agentic systems — including golden datasets, test cases, output validation, hallucination detection, and regression testing
• Establish and maintain evaluation metrics for agent performance, reliability, and clinical appropriateness; measure agent accuracy, hallucination rates, clinical validity, and real-world impact
• Implement observability, evaluation, and regression testing frameworks specific to agentic systems — decision tracing, lineage logging, and performance tracking
• Collaborate with data engineering and platform teams to integrate agent-built outputs (dbt models, transformation logic, recommendations) into existing data architectures and clinical workflows
• Ensure all agentic systems comply with healthcare regulations (HIPAA, FDA guidance on AI/ML) and responsible AI practices — including explainability, auditability, and clinician trust
• Continuously evaluate new LLM models, agent frameworks, prompt engineering techniques, and tooling; recommend adoption or migration based on healthcare-specific requirements (accuracy, cost, latency, regulatory alignment)
• Partner with data engineering to establish robust data validation and input validation layers for agents — agents are only as good as the data they operate on
• Lead experimentation and measurement of AI-automated systems impact on speed, quality, compliance, and cost across healthcare workflows
• Document agent architectures, prompt strategies, evaluation frameworks, and best practices for both technical and non-technical stakeholders
• Mentor AI Connector Engineers and other team members on agentic development patterns, LLM-powered application design, and responsible AI practices
• Work on-call as needed to support production agentic systems, troubleshoot agent issues, and respond to performance degradation or hallucination detection

Re…