Senior QA Engineer – Azure AI

We are seeking a highly skilled Senior QA Engineer specializing in Azure AI to join our client’s team in Doha (on-site). This role focuses on ensuring the reliability, accuracy, and safety of AI/LLM systems deployed on Azure, with a strong emphasis on evaluation frameworks, automation, and modern GenAI architectures.

Location:

Doha, Qatar — On-site
Role Type:
Full‑Time, Permanent
Language

Required:

English

• Design and execute QA strategies for LLM, Generative AI, and RAG‑based systems, ensuring reliability, correctness, and stability.
• Build automated evaluation pipelines using Azure Dev Ops, integrating model testing into CI/CD workflows.
• Develop advanced Python‑based testing frameworks, including Pandas workflows, PyTest suites, and custom evaluation scripts.
• Validate AI behaviours using both deterministic assertions and LLM‑as‑a‑Judge evaluation methodologies.
• Collaborate with data engineering and platform teams to maintain evaluation datasets, including ground truth / golden datasets.
• Test and validate Azure AI deployments end‑to‑end, including Azure AI Studio, Azure OpenAI, Azure AI Search, and related services.
• Contribute to Data Ops processes and continuous monitoring of AI system performance, regression, and drift.
• Support experimentation workflows, ensuring reproducibility, traceability, and quality across multiple model versions.
• Stay updated on emerging GenAI methodologies, including agentic architectures, multi‑agent workflows, and evaluation best practices.

• 1–2 years of hands‑on Generative AI experience.
• Practical expertise with RAG architectures (Vector Search + LLM).

• 5+ years of Azure experience.
• Proven deployment/testing experience in Azure AI Studio or Azure OpenAI Service (not just local testing).

• 7+ years of Python experience.
• Ability to write complex, production‑quality code (Pandas, PyTest, automation frameworks).

• 8+ years of QA experience.
• Strong understanding of LLM evaluation metrics and frameworks.

• 5+ years working with Data Ops or CI/CD pipelines.
• Direct experience with Azure Dev Ops pipelines tied to AI evaluations.

• Technical degree in Computer Science, Engineering, Mathematics, or equivalent professional experience.

• Experience with Agentic AI (Lang Graph, Auto Gen) or multi-agent systems.

• Experience with Azure AI Search / Cognitive Search including indexing and vector optimization.

• Experience with Lang Chain or Semantic Kernel.

• Experience building evaluation pipelines using Prompt Flow.

• Experience creating and managing Golden Datasets.

• Master’s degree in Data Science or AI‑related field.

• Experience fine‑tuning SLMs (e.g., Phi‑3, Llama).

• Familiarity with Azure ML, Work spaces, and Compute management.

• Background in C#/.NET for backend integration.

• Exposure to Red Teaming and adversarial testing.

• Experience with MLflow.

• Certification:
  Azure AI Engineer Associate (AI‑102).