Applied Science Manager, Artificial General Intelligence , Quality Automation

Applied Science Manager

The Artificial General Intelligence (AGI) team is seeking a dedicated, skilled, and innovative Applied Science Manager with a robust background in machine learning, statistics, quality assurance, auditing methodologies, and automated evaluation systems to lead a team ensuring the highest standards of data quality, to build industry-leading technology with Large Language Models (LLMs) and multimodal systems.

Key Responsibilities:

• Lead and mentor a team of Applied Scientists who develop comprehensive quality strategies and auditing frameworks that safeguard the integrity of data collection workflows.
• Guide the team in designing auditing strategies with detailed SOPs, quality metrics, and sampling methodologies that align with core scientist team developing Amazon Nova models.
• Oversee expert-level manual audits, meta-audits to evaluate auditor performance, and provide coaching to uplift overall quality capabilities across the team.
• Manage the development and maintenance of LLM-as-a-Judge systems, including designing judge architectures, creating evaluation rubrics, and building machine learning models for automated quality assessment.
• Oversee the configuration of data collection workflows and ensure effective communication of quality feedback to stakeholders.
• Have a direct impact on enhancing customer experiences through high-quality training and evaluation data that powers state-of-the-art LLM products and services.
• Recruit, hire, and develop team members, conduct performance reviews, set clear expectations and growth plans, and foster a culture of scientific excellence and innovation.
• Communicate with senior leadership, cross-functional technical teams, and customers to collect requirements, describe product features and technical designs, and articulate product strategy.

A day in the life:

• Lead quality solution design, guide root cause analysis on data quality issues, drive research into new auditing methodologies, and find innovative ways of optimizing data quality while setting examples for the team on quality assurance best practices and standards.
• Work closely with talented engineers, domain experts, and vendor teams to put quality strategies and automated judging systems into practice.
• Conduct regular 1:1s with team members, provide mentorship and coaching, and ensure the team delivers high-impact results aligned with organizational goals.