AI Agent Performance Engineer

AI Agent Performance Engineer About Basata

Location:

Tempe, AZ or San Francisco, CA (in-office)

At Basata, we’re rethinking the way healthcare practices get work done. Our AI-powered tools help automate the repetitive, time consuming tasks that bog teams down, so staff can spend less time on admin and more time with patients.

We’re a small, fast-moving team that loves building smart systems and simple, intuitive experiences. If you're someone who gets excited about early-stage energy, designing better ways to work, and fixing what’s broken in healthcare, we’d love to connect.

We’re looking for an AI Agent Performance Engineer to own the quality, reliability, and performance of Basata’s conversational agents. You’ll build the observability, QA, and evaluation systems that ensure our agents consistently meet accuracy, speed, and reliability benchmarks.

This role sits at the intersection of engineering, analytics, and applied AI. You’ll own and track the performance metrics that matter, detect drift or degradation early, and partner closely with Product and AI/ML engineering to drive measurable improvements in agentic behavior.

Your work will directly shape how Basata delivers high-quality AI automation to every practice we support.

• Build and maintain observability and monitoring systems that track agent latency, accuracy, call outcomes, call-flow behavior, and failure modes in production.

• Develop rigorous QA and regression testing frameworks for Voice AI agents, including automated evaluation pipelines and scripted test scenarios.

• Define, instrument, and own key performance metrics and ensure agents consistently meet or exceed them.

• Analyze large volumes of agent interactions to identify trends, misclassifications, drift, prompt issues, or reliability gaps.

• Partner closely with Product, AI/ML, and Deployment teams to convert findings into concrete product, prompt, and architecture improvements
  .

• Build internal dashboards, alerts, and reports that make agent performance transparent for internal teams and customers.

• Conduct root-cause investigations for production issues and drive sustainable, long-term fixes.

• Document QA processes, performance baselines, and best practices to support fast scaling across specialties.

• 3–5+ years of experience in software engineering and Applied AI.

• Hands-on experience building, deploying, and maintaining LLM-based agentic applications in production is essential.

• Strong working knowledge of agentic AI frameworks

• Experience building observability, monitoring, or QA systems using logs, metrics, dashboards, or automated evaluation pipelines.

• Proficiency with Python, analytics tooling
  , and comfort working with large-scale data logs.

• Experience in fast-paced, zero-to-one environments and comfort operating with autonomy.

• Drive real impact. Your work will help clinics run more smoothly and ensure patients get the care they need—faster and with fewer headaches.

• Shape something meaningful. From early product decisions to UI details, you'll play a big role in crafting both the code and the overall user experience.

• High ownership. Join a team where you’re trusted to lead, build, think critically, and bring ideas to life.

• Work with purpose. We’re not here to throw more tech at the wall, we're solving real problems in healthcare with tools that people rely on every day.

• Find a way: We own outcomes—no excuses, no shortcuts. When obstacles appear, we remove them. When the path isn’t clear, we forge one. We take extreme ownership, stay scrappy, and do what it takes to get the job done. Whatever the challenge, we find a way.

• Dream big: We refuse to accept that healthcare has to be broken. We dream boldly and act deliberately to build a future where healthcare organizations are free to focus on what truly matters: caring for people. We challenge what’s possible, rethinking every mundane barrier standing in the way of better care. Our vision is ambitious—some might call it impossible—but we believe patients and providers deserve nothing less.

• Obsess over the customer: We put our customers at the center of everything we do. We…