AI Production Analytics & Observability Lead

• Validate readiness of Agentic AI use cases for production deployment

• Track deployment success metrics and post-production performance

• Identify gaps between expected vs. actual outcomes

• Define metrics for:
Accuracy and response quality, Task completion success rates, Hallucination and failure cases, Latency and throughput

• Build evaluation datasets and validation pipelines

• Analyze:
Agent workflows and decisions, Prompt-response chains, Tool usage and orchestration behavior

• Develop observability dashboards using telemetry and logs

• Detect and escalate production issues and anomalies

• Perform root cause analysis on failures and performance issues

• Deliver executive-level reporting on AI system effectiveness

• Provide actionable insights to improve system design and outcomes

• Work closely with:
Lead Architects for feasibility alignment AI/ML engineers for model/system improvements Product teams for use case refinement

• Translate technical findings into clear business insights

Requirements

• 6-8 years experience in data analytics, data science, or AI Systems analysis or similar role required
• Experience supporting AI/ ML or GenAI systems in production environments preferred
• Auto finance experience preferred, cross functional Agile team experience preferred
• Bachelor’s Degree in Data Science, Computer Science, Engineering or related quantitative field Preferred
• Master’s Degree in related quantitative field Preferred

🔍 ATS Optimization Keywords
Below are skills and terms extracted directly from this job posting to improve Applicant Tracking System (ATS) visibility. This unique feature helps candidates tailor their applications more effectively — a feature exclusive to Job Tailor job listings.

Hard Skills

• data analytics
• data science
• AI systems analysis
• metrics definition
• root cause analysis
• evaluation datasets
• validation pipelines
• observability dashboards
• telemetry
• logs

Soft Skills

• communication
• collaboration
• problem-solving
• analytical thinking
• reporting
• insight generation
• cross-functional teamwork
• feasibility alignment

Certifications & Qualifications

• Bachelor’s Degree in Data Science
• Bachelor’s Degree in Computer Science
• Bachelor’s Degree in Engineering
• Master’s Degree in related quantitative field