AI Product Analyst

AI Product Analyst

Our client is building cutting-edge AI products for their customers and for colleagues across the group, in tax, HR, health and safety, and care. We can see that people use them; what we cannot yet see with any precision is how well they perform – for which clients, in which sectors, on which kinds of question – and therefore where to invest next.

This role answers that. You will own product analytics for our AI products: measuring performance across the dimensions that matter, building the reporting and dashboards the team and the business work from, and drawing out the insight that sits behind the figures. The purpose is to give the business a reliable evidence base for its decisions about where to invest.

You will also own the evaluation programme. Where analytics establishes how the product performs in the hands of real users, evaluation establishes whether a change is good enough to release. You will own the datasets the product is tested against and the analysis of what the results mean, working closely with our AI Domain & Workflow Specialist.

This is a fantastic opportunity to join a respected, fast-growing AI division based full-time out of our London office. You’ll work alongside a highly technical engineering team to shape what ships, rather than just describing it after the fact.

• Product Performance & Analytics: Measure how our AI products perform across retrieval quality, correctness of output, and user engagement.

• Observability Management: Own production quality observability, tracking metrics like thumbs-down rates, regeneration rates, task abandonment, and usage drift.

• Dashboard Engineering: Build and maintain the daily analysis and dashboards the AI team relies on, working with engineers to instrument and capture missing data points.

• Strategic Insight & Segmentation: Interrogate data using cohort comparison, trend analysis, and segmentation (by client type, subject matter, and question type) to identify high-material patterns and outliers.

• Evaluation Programme Ownership: Define what gets tested, analyse evaluation outputs for regressions, and collaborate with engineers to integrate testing harnesses into the build pipeline.

• Golden Dataset Curation: Build, version, and maintain curated sets of questions and known‑good answers, running elicitation sessions with subject‑matter experts to capture professional edge cases.

• Cross-Functional Reporting: Translate complex findings into actionable recommendations for the Product Manager and Director to shape priorities and inform the broader roadmap.

We’re looking for AI product analyst who combine strong technical knowledge with excellent communication skills, a proactive approach and a genuine focus on client service.

You’ll ideally have:

• Substantial experience as a product, data, or commercial analyst, having owned end-to-end product measurement.

• Strong hands‑on SQL and Python skills (or close equivalents), with the ability to construct your own automated analysis and dashboards completely independently.

• Comfort working with AI or ML systems, possessing a firm understanding of why measuring probabilistic models differs from measuring deterministic products.

• Exceptional stakeholder communication skills, specifically the ability to draw structured data requirements out of subject‑matter experts and hold evidence‑based recommendations in front of senior leaders.

• Familiarity with advanced AI concepts (Desirable):
  Experience with RAG (Retrieval-Augmented Generation) systems, golden datasets, scoring, regression testing, or A/B experiment design.

• Domain Exposure (Desirable):
  Prior experience in tax, HR, health and safety, care, or another regulated professional services environment.
  
  Salary &…