Senior AI Test Automation Engineer

• Engage directly with users and stakeholders to collect feedback, observe real usage patterns, and identify consistent user flows through LLM-powered features
• Translate user feedback and production traces into curated evaluation datasets in Lang Smith
• Design, build, and maintain offline evaluation suites for regression, benchmarking, and backtesting
• Develop and calibrate heuristic/code-based checks, LLM-as-judge evaluators, and pairwise comparisons
• Validate evaluator reliability against human review
• Instrument and maintain end-to-end tracing across Lang Graph agents and workflows using Lang Smith
• Manage annotation queues and human-in-the-loop feedback workflows
• Analyze agent trajectories and multi-step Lang Graph executions to identify failure points and distinguish LLM variance from product defects
• Integrate evaluation runs and automated tests into CI/CD pipelines as quality gates
• Support production auditing, online evaluations, quality drift detection, and alerting
• Design, develop, and implement automated UI, API, integration, regression, and deterministic E2E tests for LLM-powered application surfaces
• Collaborate with QA, development, product, and business teams to translate requirements, acceptance criteria, and expected agent behaviors into test and evaluation coverage
• Analyze and triage automation test failures
• Support test data management and test-environment stability
• Report test results, evaluation outcomes, coverage, quality trends, and risks
• Participate in code reviews and contribute to automation and evaluation standards, reusable components, and best practices
• Engage with the QA Center of Excellence to align automation and AI evaluation practices with enterprise standards

• Must be eligible to work in the US without Visa Sponsorship
• 5+ years of experience in test automation engineering, software quality assurance, or a related role
• 1–2+ years of hands‑on experience testing or evaluating LLM-powered applications
• Hands‑on experience with Lang Smith for tracing, evaluation, and observability of LLM applications
• Hands‑on experience with Lang Graph, including graph‑based agent workflows, state and context management, and tool calling
• Hands‑on experience building and maintaining automated test suites using Playwright, preferably with Type Script
• Working proficiency in Python and/or Type Script
• Experience automating UI and API testing, and experience with API testing tools or frameworks
• Familiarity with CI/CD tools such as Azure Dev Ops, Jenkins, Git Hub Actions, or equivalent
• Working knowledge of Git
• Understanding of Agile/Scrum delivery practices and participation in Agile ceremonies
• Ability to analyze requirements and user feedback, identify test and eval scenarios, and create maintainable automated coverage
• Strong problem‑solving, troubleshooting, communication, and collaboration skills
• Ability to understand and document end‑to‑end business processes, user journeys, and operational workflows
• Experience with online evaluations, production monitoring, and quality drift detection for LLM applications
• Experience with agent trajectory evaluation, RAG evaluation, or guardrails validation
• Familiarity with prompt engineering and prompt versioning workflows
• Understanding of statistical approaches to nondeterministic testing
• Experience authoring BDD/Gherkin scenarios using Cucumber or a similar framework
• Experience with contract testing tools
• Familiarity with SAFe and Agile Release Train (ART) practices
• Experience with AI‑assisted engineering practices
• Experience with performance, accessibility, mobile, or security test automation
• Experience with test management and defect‑tracking tools, such as Azure Dev Ops

Demonstrates expertise in test automation engineering and quality assurance for LLM-powered applications, with a strong focus on developing and maintaining automated test suites, analyzing user feedback, and ensuring product quality through rigorous evaluation practices.

• Test Automation Engineering
• Lang Smith Experience
• Lang Graph Workflows
• Automated Test Suites Development
• CI/CD Tools Familiarity

• Test Automation Engineering
• LLM Application Evaluation
• Automated UI Testing
• API Testing
• Python Programming
• Type Script Programming
• BDD/Gherkin Scenarios
• Statistical Testing Approaches
• Performance Test Automation
• Accessibility Test Automation

• Problem‑Solving
• Troubleshooting
• Communication
• Collaboration

• Agile Practices
• Scrum
• Quality Drift Detection
• Human‑in‑the‑Loop Workflows
• Production Monitoring

• Lang Smith
• Lang Graph
• Playwright
• Azure Dev Ops
• Jenkins
• Git Hub Actions
• Cucumber
• Test Management Tools