AI Product Analyst

This role sits within a centralized AI Engineering and Automation Center of Excellence responsible for delivering enterprise-scale AI and automation solutions across a large, distributed North American organization.

The AI work is real, deployed, and actively supporting business operations. This is not a pilot program or innovation lab.

Current capabilities include:

• Reusable AI skills and plug-ins packaged into governed applications used across multiple business functions
• Retrieval-Augmented Generation (RAG), embeddings, and multi-step workflows operating on governed enterprise data
• A governed AI asset marketplace and Model Context Protocol (MCP) catalog covering intake, publishing, versioning, lifecycle management, and maintenance
• Enterprise-grade evaluation frameworks, golden-set testing, guardrails, monitoring, and logging designed to ensure reliable AI outputs at scale

The organization also operates a mature automation ecosystem alongside its AI initiatives, including:

• Approximately 125 unattended production robots
• More than 200 production automations
• More than 19,000 automation jobs executed every 30 days
• Platforms including UiPath and Microsoft Power Platform

For projects requiring significant engineering investment, such as new backend services, infrastructure implementation, or highly complex integrations, a dedicated Enterprise Engineering organization partners with the AI Engineering team to build and deliver solutions. Once completed, those assets are cataloged, governed, and managed through established operational processes.

This role serves as the Product Owner for the AI Engineering delivery pipeline
.

The successful candidate will lead and direct solution development efforts while partnering closely with engineering teams. This is not a hands‑on software engineering role
, and the individual in this position will not be writing production code.

Key responsibilities include:

• Managing intake, qualification, prioritization, and opportunity assessment for AI initiatives
• Developing business cases and defining value measurement strategies, including establishing baselines, success metrics, attribution methodologies, and measurable outcomes
• Defining requirements and acceptance criteria for AI and LLM-enabled solutions
• Authoring evaluation specifications, curating golden datasets, and establishing acceptance thresholds used to determine production readiness
• Leading user validation activities, adoption tracking, and value realization assessments after deployment
• Monitoring portfolio health across AI assets, including ownership, lifecycle management, versioning, deprecation, and retirement planning
• Managing governance processes for AI skills, plug-ins, and MCP catalog assets, translating security, privacy, compliance, and regulatory requirements into actionable product requirements
• Building strong partnerships with business leaders, operational stakeholders, and executive teams while maintaining disciplined intake and prioritization processes

Engineering teams own:

• Evaluation harness implementation
• Monitoring instrumentation

This role owns:

• Defining what success looks like
• Defining evaluation criteria
• Defining acceptance thresholds
• Determining production readiness
• Making the final product recommendation for release

Candidates who view evaluation criteria, testing strategy, or acceptance definitions as purely engineering responsibilities are unlikely to be successful in this position.

The following are required screening criteria.

Candidates who do not meet all requirements should not be considered a fit.

Three or more years of experience in:

• Product Management
• Product Ownership
• Product Analysis

Including ownership of at least one AI or LLM-enabled product successfully deployed to production.

Pilot projects, demonstrations, prototypes, and proof-of-concepts do not satisfy this requirement.

Demonstrated ownership of acceptance criteria and evaluation processes for AI or LLM-enabled functionality.

Candidates should be able to clearly explain:

• Evaluation specifications
• Acceptance thresholds

that they personally defined.

Working knowledge of:

• Retrieval-Augmented Generation (RAG)
• Faithfulness
• Groundedness
• Retrieval Precision
• Retrieval Recall
• Answer Relevance
• Regression Testing across prompt and model versions

Enough depth to guide engineering teams, evaluate outputs, and challenge results without requiring direct coding responsibilities.