Senior AI QA Engineer

We're hiring a Senior QA Automation Engineer for one of the leading players in Sovereign AI, a UAE-based company with frontier-model access and agentic AI products live across critical sectors including large-scale construction, government, healthcare, and defense.

This is not traditional QA. You'll own quality for AI and agentic systems, defining what "good" looks like when a system reasons, calls tools, and can be wrong in subtle ways. If you've figured out how to test something whose output is different every time, this is for you.

• Test agentic systems end to end: tool-use and trajectory evaluation, grounding and citation verification, honest-failure and refusal calibration, guardrail and adversarial testing (prompt injection, jailbreaks)
• Build evaluation for non-deterministic AI: scoring for hallucination, consistency, and accuracy against gold datasets, including LLM-as-judge calibrated against human judgement
• Own prompt and model regression suites that catch quality drift before it ships
• Run continuous evaluation on live production traffic, with quality monitoring, alerting, and incident response
• Cover the fundamentals too: integration and API testing, secure gateway validation (RBAC, PII), and performance/load testing

• 5+ years in QA or quality engineering, with 2+ years testing ML, LLM, or AI agent systems
• Hands-on with LLM/agent evaluation frameworks (Deep Eval, Tru Lens, RAGAS, or custom Python evaluators) and LLM-as-judge
• Agent observability and tracing (Lang Smith, Langfuse, Open Telemetry)
• Expert Python (Pytest, custom harnesses), plus Selenium/Playwright and API testing

• Abu Dhabi hybrid, contract-based
• Onboarding ASAP
• World-class environment: elite talent from FAANG and frontier AI labs