Senior Machine Learning Engineer

About the Role

Evaluation is the bottleneck in healthcare AI— you can’t ship what you can’t measure. You’ll build the systems that tell us whether our models are safe, accurate, and ready for real patients: evaluation frameworks, synthetic data pipelines, automated benchmarks, and LLM-as-judge systems. This is a high‑leverage engineering role where your work directly gates what goes to production.

• Design and build evaluation frameworks for LLM safety, clinical accuracy, and conversational quality
• Develop synthetic data generation pipelines to stress-test models across diverse clinical scenarios
• Build automated and human‑in‑the‑loop evaluation pipelines at scale
• Create benchmarks, metrics, and LLM-as-judge systems for healthcare tasks and conversational experience
• Analyze failure modes and translate findings into actionable model improvements by collaborating with the LLM post‑training team
• Collaborate with research, engineering, and clinical teams to define and raise the quality bar

• MS or PhD in CS or related field
• 5+ years in ML engineering, evaluation systems, or applied ML
• Strong software engineering skills—Python, PyTorch, and production‑quality code
• Hands‑on experience with LLM evaluation, benchmarking, or synthetic data generation
• Comfort building robust data analysis and evaluation infrastructure, not just running experiments
• Experience with UI/UX and front‑end development toolkits such as Streamlit, Gradio, React, etc.

• Experience in healthcare AI
• Experience with RL/RLVR/RLHF or safety evaluation

We believe the best ideas happen together. To support fast collaboration and a strong team culture, this role is expected to be in our Palo Alto office five days a week, unless otherwise specified.

Please be aware of recruitment scams impersonating Hippocratic AI. All recruiting communication will come from  email addresses. We will never request payment or sensitive personal information during the hiring process.