Technical Product Manager, Data Science

CENTRL is a leading risk and compliance technology company that provides AI powered enterprise-grade risk, due diligence, cyber security and privacy management solutions to financial institutions worldwide. Our clients include some of the largest banks and investment management firms across the Americas, Europe and APAC. Headquartered in Silicon Valley, CENTRL has regional offices in New York, India, Australia, and the United Kingdom.

Established in 2015, CENTRL is a high-growth, venture backed SaaS firm leading the way in innovative generative AI solutions to manage third party risk and due diligence.

We are looking for an owner for the accuracy, reliability, and cost-efficiency of the AI behind CentrlX, our agentic platform for Manager Research and Investor Relations teams at investment firms.

We have a strong engineering team building the platform: agents, skills, knowledge, automations, connectors, and governance. We are looking for a single owner for output accuracy and efficiency.

You will own the model and prompt layer end to end. You will build the evaluation datasets and harnesses that tell us whether a change actually helped, run structured comparisons across model providers on accuracy, latency, and cost, and drive the resulting changes into the product. You will help create and design agents and skills, and work closely with sales and PS in order to help drive standards for prompting and agents, and then feed that back into the product design.

You will do a real share of this work yourself from prompt iteration to writing user facing stories like incorporating interactive feedback. While we have support teams to intake client issues and fix them, you will own AI Quality and be the point person for what is happening and what needs to improve.

This is a product role, not a research role. We want the analytical rigor of a data scientist paired with the judgment of a product manager: someone who can run the experiment, interpret it honestly, and then turn it into a shipped change.

• Build and own CentrlX's evaluation foundation from the ground up: golden datasets, grading rubrics, LLM-as-judge pipelines calibrated against human labels, and regression suites that run before prompt or model changes ship.
• Define what "good" means for each core workflow — document digitization and extraction, retrieval and groundedness, Smart Summary, Smart Response, Smart Evaluation, and full multi-step agent runs — and set a measurable quality bar for each.
• Evaluate agent behavior, not just single responses: tool selection, retrieval quality, step sequencing, and whether the finished deliverable holds up to a practitioner's review.
• Turn every real client failure into a permanent eval case, so the same class of error does not come back.

• Continuously benchmark models across providers — OpenAI, Anthropic, Google, open-weight, and specialized document models — on accuracy, latency, and cost for each workflow, and make the call on what we run where.
• Own model migrations end to end, including our in-flight move off GPT-4.1 in document digitization, where current alternatives are materially faster, cheaper, and more accurate.
• Track and manage AI spend by workflow, and use routing, model tiering, caching, and context strategy to hold quality while bringing cost down.
• Maintain a working view of the model landscape — releases, pricing changes, deprecations — and turn it into a recommendation with evidence attached, not a newsletter.

• Define the logging and tracing we need — prompt inputs, retrieved context, prompt text, outputs, tool calls, token counts, latency — and write the stories to get it…