Automation Test Architect - Agentic AI

Job Summary

We are seeking a Test Automation Architect to define and lead an enterprise‑grade test automation strategy across UI, API, data, and platform layers while evolving our quality engineering capabilities to support Agentic AI / LLM‑based systems. The role involves architecting scalable automation frameworks, driving engineering excellence, and accelerating delivery through highly productive, maintainable automation programs.

• Automation Strategy & Architecture
  • Define and own the test automation architecture and multi‑layer strategy (UI/API/contract/integration/e2e) aligned to product risk and release cadence.
  • Establish automation standards, patterns, and reference implementations (framework structure, page‑object/screenplay usage, test data patterns, reporting).
  • Drive shift‑left testing practices (contract testing, component testing, pre‑merge validation) and reduce reliance on fragile end‑to‑end tests.
• Agentic AI / LLM Test Model Alignment
  • Design and implement testing approaches for non‑deterministic AI outputs (confidence/threshold‑based assertions, semantic similarity checks, rubric‑based validation, goldens/benchmark datasets, evaluation harnesses).
  • Develop hallucination and groundedness testing strategies and regression detection across model/prompt/version changes.
  • Define AI test coverage across accuracy, robustness, bias/safety (as applicable), performance, and reliability.
  • Partner with ML/AI, Product, and Engineering to establish acceptance criteria for AI capabilities and measurable quality gates.
• High‑Productivity Automation Programs
  • Build and scale high‑throughput automation projects that are reliable, fast, and easy to extend.
  • Improve test execution time, reduce flaky tests, and optimize CI/CD pipelines (parallelization, test selection, retries with guardrails).
  • Create actionable, engineering‑friendly quality dashboards (pass/fail trends, flake rate, coverage, defect escape metrics).
• Tooling, CI/CD, and Quality Gates
  • Integrate automation into CI/CD with quality gates (PR checks, nightly suites, release readiness criteria).
  • Standardize observability for testing: logs, tracing hooks, screenshots/video, artifacts, and test run diagnostics.
  • Define practices for test environments, data management, service virtualization, and dependency control.
• Technical Leadership & Mentorship
  • Serve as the QE technical authority: coach engineers on automation design, code quality, and best practices.
  • Lead design reviews for test architecture and automation PRs; enforce coding standards and maintainability.
  • Influence roadmap and delivery by translating risk into clear testing priorities.

• 8–12+ years in Quality Engineering / Test Automation with architect‑level ownership of frameworks and strategy.
• Proven experience designing scalable automation across API testing (contract + integration), UI automation with stable locator strategies, test data strategy, and environment reliability.
• Strong programming skills in at least one modern language (Java, Python, Type Script/JavaScript, C#).
• Expertise with automation tools/frameworks (UI:
  Playwright / Cypress / Selenium; API: REST‑assured / Super Test / Postman/Newman / pytest; CI: Git Hub Actions / Jenkins / Azure Dev Ops).
• Demonstrated ability to reduce flakiness and improve execution speed through engineering solutions.

• Experience testing LLM/Agentic AI features or other probabilistic systems (evaluation harnesses, benchmark datasets, semantic scoring, drift detection, prompt/version regression strategies, structured evaluation reports).
• Familiarity with modern quality approaches: test pyramid, contract testing, consumer‑driven contracts, risk‑based testing, shift‑left.
• Knowledge of security/performance testing integration (baseline understanding; hands‑on is a plus).
• Experience building reusable automation libraries for multiple teams/products.

Base salary range: $89,600.00 – $ USD. This role is eligible for a bonus. Compensation is based on the primary location of the position.

• Medical, Dental, & Vision Plans
• 401(k) with company matching
• FSA/HSA
• Commuter benefits
• Tuition assistance plan
• Vacation and sick time
• Paid parental leave

All qualified applicants will receive consideration without regard to race, color, religion, sex (including pregnancy, gender identity, transgender status, and sexual orientation), national origin, disability, age, genetic information, veteran status, or any other characteristic protected by applicable law. We do not tolerate discrimination on any of these bases.