Applied Scientist, Foundational AI

Applied Scientist

The Artificial General Intelligence (AGI) team is seeking a dedicated, skilled, and innovative Applied Scientist with a robust background in machine learning, statistics, quality assurance, auditing methodologies, and automated evaluation systems to ensure the highest standards of data quality, to build industry-leading technology with Large Language Models (LLMs) and multimodal systems.

Key job responsibilities include:

• Collaborating closely with the core scientist team developing Amazon Nova models.
• Leading the development of comprehensive quality strategies and auditing frameworks that safeguard the integrity of data collection workflows.
• Designing auditing strategies with detailed SOPs, quality metrics, and sampling methodologies that help Nova improve performances on benchmarks.
• Performing expert-level manual audits, conducting meta-audits to evaluate auditor performance, and providing targeted coaching to uplift overall quality capabilities.
• Developing and maintaining LLM-as-a-Judge systems, including designing judge architectures, creating evaluation rubrics, and building machine learning models for automated quality assessment.
• Setting up the configuration of data collection workflows and communicating quality feedback to stakeholders.
• Having a direct impact on enhancing customer experiences through high-quality training and evaluation data that powers state-of-the-art LLM products and services.

An Applied Scientist with the AGI team will support quality solution design, conduct root cause analysis on data quality issues, research new auditing methodologies, and find innovative ways of optimizing data quality while setting examples for the team on quality assurance best practices and standards. Besides theoretical analysis and quality framework development, an Applied Scientist will also work closely with talented engineers, domain experts, and vendor teams to put quality strategies and automated judging systems into practice.