Manager, AI Analytics

Location:

United States

National Debt Relief is seeking a data-savvy Manager of AI Analytics to serve as an individual-contributor analytics expert supporting the measurement, testing, and optimization of agent-replacement AI across voice and chat channels. This role will execute the analytics framework that governs AI interaction performance by designing tests, tracking KPIs, and building the dashboards that give the business real-time visibility into how AI is performing against quality, containment, and client experience goals.

Reporting to the Director of Digital & AI Analytics, this individual will be a hands-on analytical partner to AI Analytics and Engineering teams, translating complex model and interaction data into clear, actionable recommendations that inform the pace and scope of AI scale-up.

• Execute the measurement framework for AI Analytics, tracking performance of agent-replacement AI interactions across voice and chat channels.
• Design and run test analysis for AI model and prompt experiments (e.g., model version A/B tests, containment vs. escalation testing), translating results into go/no-go recommendations.
• Build and maintain dashboards and scorecards tracking AI-handled interaction KPIs — containment rate, accuracy, quality/CSAT, and escalation rate.
• Partner with AI Analytics and Engineering teams to ensure AI interaction data (transcripts, model outputs, outcomes) flows cleanly into unified reporting.
• Analyze AI-handled interaction volume and performance trends to support decisions on scaling AI across channels, balancing performance thresholds and client experience.
• Collaborate with data engineering to validate and maintain data flows into Snowflake for AI analytics use cases.
• Communicate findings and recommendations clearly to both technical and non-technical stakeholders, including Analytics leadership.
• Identify data quality issues or gaps in AI interaction tracking and drive resolution with cross-functional partners.
• Stay current on conversational AI/LLM measurement practices and evolving best practices in AI performance analytics.

• Bachelor's degree in Business, Economics, Data Science, or related field required.
• 6 years of experience in analytics, business intelligence, or similar roles.
• Experience in financial services or fintech, especially client onboarding or customer support journeys preferred.
• Previous exposure to leading or mentoring junior analysts, even without formal management experience preferred.

• Strong SQL skills (Snowflake preferred) and comfort with data modeling and manipulation.
• Experience analyzing conversational AI / LLM-driven interactions (chat or voice bots), including containment, accuracy, and quality metrics.
• Experience with A/B testing frameworks and experimentation design, ideally for AI models or prompts.
• Experience with visualization and dashboarding tools like Tableau, Sigma, Looker, or Power BI.
• Ability to communicate complex data clearly and drive action across technical and non-technical teams.
• Ability to consolidate, understand, and communicate large data sets independently.
• Familiarity with server-side tracking, CDPs (e.g., Segment, mParticle), or customer data governance frameworks preferred.
• Experience with Python, Big Query ML, or Airflow is a plus.

National Debt Relief Role

Qualifications:

• Computer competency and ability to work with a computer.
• Prioritize multiple tasks and projects simultaneously.
• Exceptional written and verbal communication skills.
• Punctuality expected, ready to report to work on a consistent basis.
• Attain and maintain high performance expectations on a monthly basis.
• Work in a fast-paced, high-volume setting.
• Use and navigate multiple computer…