Senior AI Eval & Safety Engineer

We are partnering with a leading financial institution in the UAE that is investing heavily in Artificial Intelligence and Generative AI technologies.

We are looking for an experienced Senior AI Evaluation & Safety Engineer to join on a 12-month contract, supporting the development, evaluation and safe deployment of enterprise-scale AI applications.

Key Responsibilities LLM Evaluation & Testing

• Design, develop and maintain automated evaluation frameworks for LLM-powered applications.
• Evaluate AI systems for accuracy, relevance, faithfulness, hallucinations, consistency and reliability.
• Develop golden datasets, evaluation benchmarks and automated testing pipelines.
• Implement LLM-as-a-Judge methodologies and automated scoring mechanisms.
• Define evaluation metrics, acceptance criteria and quality standards for production AI applications.
• Integrate evaluation and regression testing into AI development workflows.

RAG & Agentic AI Evaluation

• Develop evaluation frameworks for Retrieval-Augmented Generation (RAG) applications.
• Assess retrieval accuracy, context relevance, answer faithfulness and response quality.
• Evaluate AI agents, including tool calling, multi-step workflows, task completion and reliability.
• Identify failure scenarios and recommend improvements to AI application performance.
• Collaborate with AI engineering teams to optimise system accuracy and reliability.

AI Safety, Security & Red Teaming

• Design and execute AI safety tests to identify vulnerabilities and unsafe model behaviour.
• Conduct adversarial testing, including prompt injection, jailbreak attempts and manipulation of AI responses.
• Evaluate risks relating to sensitive data exposure, confidential information leakage and unauthorised actions.
• Develop and implement AI guardrails, safety controls and validation mechanisms.
• Support AI red teaming exercises and security assessments.
• Ensure AI applications align with responsible AI and enterprise security standards.

Evaluation Frameworks & Automation

• Build scalable evaluation pipelines using Python and relevant LLM evaluation frameworks.
• Work with tools such as RAGAS, Deep Eval, Lang Smith, Tru Lens, promptfoo or equivalent technologies.
• Develop automated testing suites to evaluate AI applications across different models and configurations.
• Monitor production AI applications for performance degradation, safety issues and unexpected behaviour.

AI Governance & Risk Management

• Support the implementation of responsible AI standards and evaluation controls.
• Ensure AI applications meet internal security, privacy and regulatory requirements.
• Document evaluation methodologies, test results, identified risks and remediation actions.
• Collaborate with cybersecurity, risk, compliance and governance teams.
• Contribute to the development of AI evaluation and safety best practices.

Required Skills & Experience

• Bachelor's or Master's degree in Computer Science, Artificial Intelligence, Machine Learning, Software Engineering or a related field.
• Strong professional experience in AI/ML engineering, applied AI or software engineering.
• Advanced Python programming skills
  , including experience developing automated evaluation and testing frameworks.
• Hands-on experience building or evaluating production Generative AI applications.
• Strong understanding of LLMs, RAG architectures, embeddings, vector databases and agentic AI systems
  .
• Practical experience implementing LLM evaluation metrics, including hallucination detection, faithfulness, relevance and response quality.
• Experience with evaluation frameworks such as RAGAS, Deep Eval, Lang Smith, Tru Lens or equivalent
  .
• Experience with AI safety testing, adversarial testing, prompt injection detection or LLM red teaming.
• Understanding of LLM security vulnerabilities, guardrails and responsible…