QA Automation Engineer

QA Automation Engineer - AI & Agentic Systems

Level: Senior IC
Location: Remote

We are building agentic AI systems that can interpret complex data sources, documentation, and structured information to perform analysis, validation, and decision-support tasks. This role exists to make those agents trustworthy enough to act on.

This is a senior, hands-on quality role weighted toward testing non-deterministic AI and agentic systems. That is where most of your time sits and where the hiring bar is highest. You will also own the broader quality surface: integration, API, and performance testing are part of the remit, not out of scope.

The differentiator for this role is the ability to define what "good" looks like when a system reasons, calls tools, and can be wrong in subtle ways. You will scope, build, and run the frameworks yourself, with the independence of a senior engineer, and decide what to build first.

• Tool-Use & Trajectory Evaluation: Test whether agents select the right tools for the right reasons and follow sound multi-step trajectories, not just whether the final answer looks plausible. Evaluate planning, intermediate steps, and recovery when a tool fails or returns nothing.
• Grounding & Citation Verification: Verify that agent claims are backed by cited evidence in source data, documents, or knowledge bases, and that referenced locations actually support the answer, catching confident but unsupported outputs.
• Honest-Failure & Refusal Calibration: Assert that agents ask for clarification or respond appropriately when information cannot be confidently determined, rather than inventing answers.
• Guardrail & Adversarial Testing: Probe prompt injection, jailbreaks, and instructions hidden within ingested content, ensuring the agent treats source content as untrusted data.
• Multi-Turn & State: Validate follow-ups, references to prior turns, and that conversational state carries correctly across a session.
  AI & LLM Validation

• Non-Deterministic Testing: Architect automated frameworks that score generative AI outputs for hallucination, consistency, and factual accuracy against gold-standard datasets using LLM-as-judge methods calibrated against human judgement.
• Prompt & Model Regression: Design regression suites that catch prompt drift and model-version drift, ensuring changes to models or system instructions do not quietly degrade quality. Own the ground-truth and evaluation datasets these depend on.

• Continuous Evaluation: Extend evaluation beyond pre-release into production, continuously scoring live agent outputs so quality is measured on real usage, not only in test environments.
• Monitoring & Alerting: Build quality monitoring that flags regressions, drift, and anomalous agent behaviour before users discover them.
• Quality Incident Response: Triage quality incidents and trace failures back to specific model versions, prompts, or datasets, feeding fixes into the development cycle.

• Backend, UI & API Testing: Build robust integration tests that validate API integration across services and key user-facing flows.
• Secure Gateway Validation: Automate testing of secure API gateways, verifying that Role-Based Access Control (RBAC) and PII-redaction logic work correctly before data reaches AI models.
• Performance & Load: Own performance test plans and implementation (for example, Locust, JMeter, k6), validating latency, throughput, and stability under realistic load.

• Data Validation: Use SQL and data-validation tooling (for example, Great Expectations) to verify data quality across data platforms and vector databases, including the ground-truth and retrieval corpora that agents depend on.
• Requirements Traceability: Map test and evaluation cases to system requirements and user needs, producing verification-and-validation evidence and quality reports needed to ship with confidence.
• Quality Gates: Enforce quality gates in Git Lab CI/CD that prevent non-compliant models or code from merging and prepare readiness evidence for release reviews.

• AI Evaluation (Core): Hands-on experience with LLM/agent evaluation frameworks (e.g., Deep Eval, Tru Lens, RAGAS, or custom Python evaluators) and LLM-as-judge techniques.
• Agent Observability: Experience tracing and debugging agent runs, including tool calls, intermediate steps, token usage, and latency, using tools such…