QA AI Automation Engineer - LLM And Model Evaluation

Location: Abu Dhabi, United Arab Emirates

Working Pattern: On-site, 5 days per week

Type: Permanent, full-time

Salary: Competitive tax-free package, dependent on experience

Relocation: Full relocation support, visa sponsorship and family status available

Company: Established technology developer (name shared at first conversation)

Tektora are supporting an established technology client in Abu Dhabi who are looking to speak with QA AI Automation Engineers for a permanent opportunity owning the testing strategy for their internal AI and LLM models.

This is not a conventional QA role, and it is not a role about using AI to test software.
The thing under test here is the AI itself. You will own how the company proves its own models behave — output quality, accuracy, consistency, edge cases and failure modes — across a mix of real production data and synthetic test data, and you will build the Python frameworks that do it from scratch.

Relocation to Abu Dhabi is required. This is an on-site environment and the role cannot be performed remotely. The package is tax-free, relocation is fully supported, and the client has moved engineers and their families into this programme before.

This role would suit

A Python-strong engineer who has moved into AI and LLM evaluation — someone who has built eval harnesses rather than only consumed them, who has worked with golden datasets, regression suites and quality gates on model and prompt changes, and who is comfortable arguing about what “good output” actually means for a given use case. You might currently call yourself a QA automation engineer, an ML or MLOps engineer, an AI evaluation engineer or a data engineer;

the title matters far less than having genuinely tested model behaviour in production.

Background can be commercial software, AI product, data platform or research engineering. Defence or national-security experience is welcome but is not required for this role.

• Building and maintaining Python automation test suites and frameworks from scratch that continuously validate the company’s internal AI models
• Testing LLM outputs for quality, accuracy, consistency, edge cases and failure modes
• Building and managing test datasets that combine real production data with synthetic data — generated scenarios, edge cases and adversarial inputs — to maximise coverage
• Writing automated test scripts that run on every model and prompt update
• Running regression tests, golden-dataset evaluations and performance benchmarks across latency, cost and throughput
• Integrating evaluation into CI/CD with quality gates that block deployment when thresholds drop
• Tracking and reporting model performance over time through dashboards and alerting so regressions are caught early
• Working with the AI, ML and product teams to define what good output looks like per use case and encoding that into automated checks

• Bachelor’s or Master’s degree
• Strong Python — scripting, testing frameworks such as pytest, and working with APIs
• Experience testing AI or LLM outputs, or comparable ML system validation, using evaluation frameworks such as RAGAS, Deep Eval or Lang Smith, or custom scoring of your own
• Practical data handling — cleaning real data, generating and synthesising test data, and building golden datasets
• Familiarity with CI/CD pipelines such as Git Lab CI or Git Hub Actions
• Basic SQL and No
  
  SQL alongside working data analysis
• Experience with FastAPI, Docker and Linux

• Working knowledge of vector search and embeddings
• Model fine-tuning concepts, prompt engineering or observability tooling
• MCP (Model Context Protocol)
• GPU and CPU optimisation
• ISTQB CT-AI certification

Interested?

Tektora are looking to speak with QA AI Automation Engineers and AI evaluation engineers who are open to a permanent move to Abu Dhabi and keen to own model quality end to end.