Senior AI Tools Testing Engineer

• hybrid
• Competitive salary
• Opportunity for advancement

Job Title: Senior / Lead Agentic AI Engineer – LLM Evaluation & Observability

Location: Dallas, TX (Hybrid)

Duration: Long‑Term

Experience: 8+ years

Interview: Client round, in‑person

Important: This is a hands‑on engineering role. Candidates with practical experience building and evaluating AI applications are preferred over those whose experience is limited to AI strategy, architecture documentation, conventional QA automation, or using AI coding assistants.

We seek a Senior/Lead AI Engineer with experience building, orchestrating, monitoring, and evaluating production‑grade Generative AI and Agentic AI applications. The ideal candidate is expert in Amazon Bedrock, Agent Core, Lang Graph, Lang Chain, RAG, LLM observability, and evaluation frameworks. The role focuses on creating reliable, scalable, production‑ready AI applications, emphasizing agent orchestration, AI testing, LLM evaluation, and continuous quality monitoring.

The candidate will collaborate with Engineering, QA, and platform teams to establish best practices for building, testing, monitoring, and improving AI‑powered applications. The testing focus is on GenAI evaluation, RAG pipeline assessment, agent behavior validation, LLM quality benchmarking, and automated evaluation frameworks, not conventional UI test automation.

Key Responsibilities 1. Agentic AI & GenAI Engineering

• Design, develop, and deploy production‑grade agentic and generative AI applications.
• Build and orchestrate AI agents using Lang Graph, Lang Chain, Amazon Bedrock, and Agent Core.
• Implement multi‑agent workflows, tool calling, state management, memory, and orchestration.
• Create scalable LLM‑powered applications and reusable AI components.
• Deploy agentic workflows with Amazon Bedrock Agent Core, handling runtime management, tool integration, and execution orchestration.

2. RAG & LLM Application Development

• Design and implement Retrieval‑Augmented Generation (RAG) solutions.
• Develop pipelines for document ingestion, chunking, embedding, retrieval, reranking, and context management.
• Optimize retrieval accuracy, response quality, latency, and cost.
• Apply grounding, hallucination reduction, and context‑relevance techniques.

3. LLM Evaluation & AI Testing

• Build automated evaluation frameworks with RAGAS, Deep Eval, and similar tools.
• Evaluate RAG pipelines using metrics such as Faithfulness, Answer Relevancy, Context Precision, Context Recall, and Answer Correctness.
• Implement LLM‑as‑a‑Judge techniques to assess generated responses, retrieval quality, agent behavior, and workflow execution.
• Establish evaluation datasets, benchmarks, and regression‑testing pipelines for continuous quality assessment.

3a. Agent Evaluation and Observability

• Implement end‑to‑end tracing of multi‑agent workflows, including tool invocation, agent handoffs, execution paths, and failure handling.
• Establish observability practices to detect hallucinations, reasoning failures, tool‑calling errors, and execution inconsistencies.

4. AI Observability & Monitoring

• Implement tracing, monitoring, and observability with Open Telemetry, Lang Smith, and Langfuse.
• Track LLM requests, agent execution flows, tool calls, latency, token consumption, errors, and performance.
• Develop debugging mechanisms for complex multi‑step agentic workflows.
• Create monitoring practices to detect performance degradation and operational issues.

5. AWS AI Platform & Production Engineering

• Deploy AI applications using Amazon Bedrock and Bedrock Agent Core.
• Integrate foundation models, agent frameworks, APIs, and enterprise data sources.
• Apply production‑readiness practices covering scalability, reliability, security, and performance.
• Support CI/CD integration and automated evaluation of AI…