QA Automation Engineer

Level: Senior IC
Location: Abu Dhabi

We are building agentic AI systems that read complex models, drawings, specifications, and regulatory documents to check compliance, plan and validate schedules, and support design.

This role exists to make those agents trustworthy enough to act on.

This is a senior, hands-on quality role weighted toward testing non-deterministic AI and agentic systems. That is where most of your time sits and where the hiring bar is highest. You will also own the broader quality surface: integration, API, and performance testing are part of the remit, not out of scope.

The differentiator we are hiring for is the ability to define what "good" looks like when a system reasons, calls tools, and can be wrong in subtle ways.

You will scope, build, and run the frameworks yourself, with the independence of a senior engineer, and decide what to build first.

• Tool-Use & Trajectory Evaluation: Test whether agents select the right tools for the right reasons and follow sound multi-step trajectories, not just whether the final answer looks plausible. Evaluate planning, intermediate steps, and recovery when a tool fails or returns nothing.
• Grounding & Citation Verification: Verify that agent claims are backed by cited evidence in the source model, drawing, or document, and that a stated location actually supports the answer, catching confident but unsupported outputs.
• Honest-Failure & Refusal Calibration: Assert that agents ask for clarification or state when something cannot be determined instead of inventing an answer, and build tests where the correct behaviour is refusal.
• Guardrail & Adversarial Testing: Probe prompt injection, jailbreaks, and instructions hidden inside ingested documents, ensuring the agent treats source content as untrusted data.
• Multi-Turn & State: Validate follow-ups, references to prior turns, and that conversational state carries correctly across a session.

• Non-Deterministic Testing: Architect automated frameworks that score generative-AI outputs for hallucination, consistency, and factual accuracy against gold-standard datasets, using LLM-as-judge methods calibrated against human judgement.
• Prompt & Model Regression: Design regression suites that catch prompt drift and model version drift, so changes to models or system instructions do not quietly degrade quality. Own the ground-truth and evaluation datasets these depend on.

• Continuous Evaluation: Extend evaluation beyond pre-release into production, continuously scoring live agent outputs so quality is measured on real traffic, not only in the test environment.
• Monitoring & Alerting: Build quality monitoring that flags regressions, drift, and anomalous agent behaviour in production before users or customers do.
• Quality Incident Response: Triage quality incidents and close the loop, tracing failures in AI logic back to the specific model version or dataset that caused them and feeding fixes into the development cycle.

• Backend, UI & API Testing: Build robust integration tests that validate API integration across services and key user-facing flows.
• Secure Gateway Validation: Automate testing of secure API gateways, verifying that role-based access controls and sensitive-data redaction logic work correctly before data reaches AI models.
• Performance & Load: Own performance test plans and their implementation using appropriate performance and load-testing technologies, validating latency, throughput, and stability under realistic load.

• Data Validation: Use SQL and data-validation tooling to verify data quality across data platforms and vector databases, including the ground-truth and retrieval corpora the agents depend on.
• Requirements Traceability: Map test and evaluation cases to system requirements and user needs, producing the verification-and-validation evidence and quality reports needed to ship with confidence.
• Quality Gates: Enforce quality gates in CI/CD pipelines that prevent non-compliant models or code from merging and prepare readiness evidence for stage and release reviews.

• AI Evaluation (core): Hands-on experience with LLM and agent-evaluation frameworks or custom Python evaluators, including LLM-as-judge techniques.
• Agent Observability: Experience tracing and debugging agent runs, including tool calls,…