QA AI Automation Engineer

We are hiring an experienced QA AI Automation Engineer to join a dynamic team in Abu Dhabi. This role owns the complete testing strategy by utilising a blend of real production data and synthetic test datasets to continuously evaluate AI models. It offers an exciting opportunity to work directly with modern AI technologies and internal LLM models in the UAE.

As a QA AI Automation Engineer, you will build and maintain Python based automation test suites from scratch. You will validate internal AI model output quality, accuracy, consistency, edge cases, and failure modes before and after every update. Collaborating with cross functional AI, ML, and product teams, you will establish performance metrics, integrate test suites into automated deployment pipelines, and maintain model reliability over time.

• Build and maintain Python based automation test suites and frameworks from scratch that continuously test internal AI models against real world and synthetic data
• Develop automation test suites for internal AI and LLM outputs covering accuracy, consistency, edge cases, and failure modes
• Build and manage test datasets combining real production data and generated synthetic scenarios to maximize test coverage
• Write automated test scripts including regression tests, golden dataset evaluations, and performance benchmarks for latency, cost, and throughput
• Integrate tests into CI CD pipelines with automated quality gates that stop deployments when thresholds drop
• Track and report model performance over time using clear dashboards and alerting systems to catch regressions early
• Work alongside AI, ML, and product teams to define quality criteria per use case and encode them into automated checks

• Bachelors or Masters Degree
• Strong Python expertise including scripting, testing frameworks such as pytest, and API integrations
• Proven experience testing AI and LLM outputs or validating ML systems using evaluation frameworks like RAGAS, Deep Eval, Lang Smith, or custom scoring
• Practical experience handling data, including cleaning real data, generating test data, and constructing golden datasets
• Hands on experience with CI CD pipelines including Git Lab CI or Git Hub Actions
• Solid understanding of SQL, No
  
  SQL databases, data analysis, and vector search or embeddings Familiarity with FastAPI, Docker, Linux, Model Context Protocol, and GPU or CPU optimizations
• Understanding of model fine tuning concepts, prompt engineering, and observability tools Strong attention to detail, conceptual thinking, initiative, and a results oriented approach ISTQB CT AI certification is a plus