AI Engineer Sr

We are looking for an AI Engineer - Agentic Systems Evaluation to help define how we measure, test, and improve the quality of enterprise AI systems.

As AI evolves from conversational assistants and Retrieval-Augmented Generation (RAG) into tool using agents, multi-agent systems, and autonomous business workflows, evaluating only the final response is no longer enough.

An agent may reach the right answer while choosing the wrong tool, taking unnecessary steps, retrieving incorrect context, failing to escape to a human or violating business rules.

This role will build the evaluation frameworks needed to understand not only whether an AI system succeeded, but how it succeeded, how reliably it can repeat that outcome, and whether the architecture is appropriate for the problem.

Build Agentic Evaluation Frameworks

Design evaluation methodologies covering the complete AI execution lifecycle:

Intent Planning Retrieval Tool Use Reasoning Action Business Outcome

Evaluate systems across dimensions including:

• task and business outcome accuracy
• planning and decision quality
• retrieval quality and groundedness
• tool selection and execution
• agent routing and delegation
• human escalation decisions
• reliability and failure recovery
• safety and policy compliance
• latency, token consumption, and cost

Evaluate Agentic Design Patterns

Design experiments and benchmarks that help engineering teams determine which architecture works best for a given problem.

Evaluate patterns such as:

• Single-agent vs. multi-agent systems
• Supervisor/router architectures
• Planner–executor patterns
• Sequential and parallel workflows
• Tool-using agents
• Human-in-the-loop workflows
• Long-running agents with state and memory

Measure whether additional agent complexity actually improves task success, reliability, and business outcomes enough to justify increased latency, cost, and operational complexity.

Advance RAG & Knowledge Evaluation

Build rigorous evaluation approaches for enterprise retrieval and knowledge systems, including:

• retrieval precision and relevance
• groundedness and citation accuracy
• source authority and freshness
• chunking, metadata, and indexing strategies
• semantic vs. hybrid search and reranking
• permission-aware retrieval

Evaluate how retrieval decisions ultimately impact downstream agent performance, rather than treating RAG evaluation as an isolated problem.

Build Automated Evaluation & Regression Testing

Develop scalable evaluation infrastructure including:

• golden and synthetic datasets
• scenario and adversarial test suites
• deterministic graders
• LLM-as-a-Judge evaluation
• human evaluation workflows
• trace-based evaluation
• automated regression testing

Integrate evaluations into AI development and release pipelines so changes to models, prompts, retrieval, tools, or agent architectures can be measured before reaching production.

Build Agent Trace & Failure Analysis

Analyze complete agent execution traces including planning, retrieved context, tool calls, handoffs, retries, exceptions, latency, and cost.

Develop failure taxonomies that distinguish between:

Model | Retrieval | Planning | Tool | Routing | Memory | Integration | Policy | Orchestration failures

Turn production failures and user feedback into measurable regression tests and engineering improvements.

Define Production AI Quality

Establish measurable quality standards and release criteria for AI systems.

Metrics may include Task Success Rate, First-Pass Success Rate, Tool Selection Accuracy, Agent Routing Accuracy, Plan Execution Fidelity, Failure Recovery Rate, Human Escalation Accuracy, Business Outcome Accuracy, Cost / Latency per Successful Task

Help teams answer a fundamental question:

Is this AI system reliable, safe, efficient, and valuable enough to operate in…