Quality Assurance Engineer

We’ve made a lot of progress since opening the doors in 1942, but one thing has never changed – our commitment to serve, heal, lead, educate, and innovate. We believe that every award earned, every record broken and every patient helped is because of the dedicated employees who fill our hallways.
At Ochsner, whether you work with patients every day or support those who do, you are making a difference and that matters. Come make a difference at Ochsner Health and discover your future today! Owns the quality engineering function for an agentic conversational AI platform — serving as the guardian of the customer experience across every voice and chat interaction. This role spans two mandates: hands‑on assurance that every release is effective and defect‑free before it reaches end users, and building, scaling, and maintaining the evaluation infrastructure that makes quality continuous, measurable, and increasingly self‑improving.

The role plans and directs the development, application, and maintenance of quality standards for conversational AI applications and their supporting systems and owns the automated evaluation platform — synthetic and adversarial agent‑to‑agent conversation testing, Deep Eval test suites, proprietary LLM-as-judge evaluators, and deterministic rule‑based checks — as the system of record for shipping model, prompt, and agent changes with confidence. Because LLM systems are non‑deterministic, this role designs test plans and evaluation rubrics that turn ambiguous notions of quality into clear, defensible metrics, and correlates those metrics with real‑world performance and user experience.

The role is weighted toward quality infrastructure and evaluation engineering (~70%), balanced with hands‑on manual and exploratory QA (~30%), with an explicit mandate to advance the platform toward recursive self‑improvement: a closed‑loop quality flywheel in which evaluation findings automatically generate new test cases, refine judge rubrics, surface model and prompt improvement candidates, and re‑validate fixes — so the evaluators, and the agents they evaluate, get better with every conversation.

To perform this job successfully, an individual must be able to perform each essential duty satisfactorily. The requirements listed below are representative of the knowledge, skill, and/or ability required. Reasonable accommodations may be made to enable qualified individuals with disabilities to perform the essential duties.

This job description is a summary of the primary duties and responsibilities of the job and position. It is not intended to be a comprehensive or all‑inclusive listing of duties and responsibilities. Contents are subject to change at the company’s discretion.

• Required – 3 years Software; quality assurance and testing methodologies; and defect‑tracking tools. Requires conversational AI testing experience (intent accuracy, fallback handling, escalation logic)
• Preferred – Familiarity with AI/machine learning and natural language processing concepts, particularly for conversational AI or chatbot testing.

• Understanding of the software development lifecycle and API/integration testing.
• Ability to design and execute detailed test cases across multiple scenarios.
• Strong attention to detail and quality standards.
• Ability to collaborate and communicate effectively with engineers and product teams to resolve defects.
• Documentation and reporting of test outcomes.
• Ability to balance speed of testing with thoroughness and accuracy while maintaining consistency and discipline in the testing process.

• Develops and implements quality standards, tests plans, and methods for inspection, testing, and evaluation across the…