AI Tester

West Yorkshire, England, United Kingdom

Company: Bit recruit Client /

Employer:

Occupop

Job reference: 82011d

We’re looking for an AI Quality Analyst to help validate and improve Generative AI products and features. This role focuses on testing LLM-based systems, prompt workflows, agent behaviours, RAG pipelines, and AI-driven user experiences. You will work closely with product, engineering, and AI teams to identify quality risks, design robust test strategies, and ensure our GenAI solutions are reliable, safe, and production‑ready.

• Design and execute test plans for Generative AI applications, including chat, summarization, extraction, classification, and agent workflows
• Validate LLM outputs for quality, relevance, accuracy, consistency, tone, and policy compliance
• Test prompt behaviour across normal, edge, and adversarial scenarios
• Evaluate RAG systems for retrieval quality, grounding, hallucination risk, and citation accuracy
• Create test cases for multi‑step AI workflows, tool use, memory, and orchestration logic
• Partner with engineers and product managers to define acceptance criteria for AI features
• Identify and document defects, regressions, safety issues, and model failure patterns
• Build and maintain test datasets, evaluation rubrics, and benchmark suites
• Support automated testing for AI features, APIs, and end‑to‑end workflows
• Track model and prompt changes to assess quality impact over time
• Contribute to release readiness by reporting on AI quality metrics, risks, and recommendations

• 3+ years in QA, software testing, or test engineering
• Hands‑on experience testing AI/ML or Generative AI applications
• Strong understanding of LLM behaviour, including hallucinations, prompt sensitivity, grounding, and non‑deterministic outputs
• Experience writing detailed test cases, bug reports, and test summaries
• Familiarity with API testing tools and test automation frameworks
• Ability to evaluate model outputs using structured rubrics and human judgment
• Strong analytical skills and attention to detail
• Comfortable working cross‑functionally in a fast‑moving product environment

• Experience testing chatbots, copilots, agents, or RAG‑based applications
• Familiarity with prompt engineering and prompt/version evaluation
• Experience with Python, SQL, or scripting for test data and automation
• Knowledge of AI safety, red teaming, bias testing, and policy evaluation
• Experience with model evaluation frameworks, telemetry, or experimentation
• Exposure to cloud AI platforms and LLM providers

• We are a multi‑award winning tech company with an aspirational mentality
• Recognitions include being named on the Reg Tech
  100 list for 2024, listed in the Top 100 Fast‑Growing Tech Companies by Northern Tech Awards 2024, and named Technology Provider of the Year by Corporate Finance Awards 2024
• Great Place To Work certified since 2022
• Excellent progression opportunities with personal development goals, regular feedback, and support
• Diverse and inclusive team committed to promoting diversity & inclusion and social responsibility, with active DE&I initiatives and community support

• 25 days holiday rising to 30 with each year of service
• Private medical insurance covering dental and optical
• Company pension scheme
• Life assurance – 4× your annual salary
• 1 day paid volunteering per year
• Enhanced maternity / paternity offerings
• Employee assistance programme
• Cycle to work scheme
• Access to a gym