Sr Applied Scientist, Support Agent Intelligence and Evaluation

Description

Amazon Advertising is one of Amazon's fastest growing and most profitable businesses. Amazon's advertising portfolio helps merchants, retail vendors, and brand owners succeed via native advertising, which grows incremental sales of their products sold through Amazon. The primary goals are to help shoppers discover new products they love, be the most efficient way for advertisers to meet their business objectives, and build a sustainable business that continuously innovates on behalf of customers.

Millions of advertisers rely on Amazon's self-service support experience to resolve issues, unblock campaigns, and grow their business. Our Support Agents team is building the science behind intelligent, conversational support — systems that understand advertiser intent, retrieve the right knowledge, generate accurate answers, and know when to elevate. We serve ~2M monthly active advertisers across dozens of locales and languages, and every percentage point of improvement in resolution quality translates directly into advertiser success and retention.

We are seeking an Applied Scientist who is passionate about building evaluation science, NLP systems, and quality measurement  will define how we measure "good" — designing LLM-as-a-judge evaluation pipelines, developing our next-generation Issue Resolution Rate (IRR) metrics, and closing the quality gap between English and non-English markets. Your work will directly shape what ships to advertisers and what leadership uses to assess the health of our support experience.

You might start the morning reviewing overnight evaluation results from your LLM-as-a-judge pipeline, then jump into a whiteboard session designing a new resolution metric that captures whether advertisers actually unblocked their campaign. After lunch you're running offline experiments on a cross-lingual retrieval model to close the quality gap for non-English markets, and by end of day you're syncing with engineering on latency trade-offs for next week's A/B test.

The constant: your science directly changes the experience millions of advertisers have when they need help.

This role sits within Amazon Advertising's broader Agentic Intelligence organization — a community of multiple Applied Science and Engineering teams building the next generation of AI-powered experiences for advertisers. You'll have access to Principal Engineers and Principal Applied Scientists to pressure-test ideas and elevate your work.

What makes this team unique is the balance: you'll drive product-facing science through customer support agents that touch millions of advertisers, while also influencing and collaborating with a core AI infrastructure team within Amazon. The team is cross-functional — scientists and engineers work shoulder-to-shoulder, from problem framing through production deployment.

• 3+ years of building machine learning models for business application experience

• PhD, or Master's degree and 6+ years of applied research experience

• Experience programming in Java, C++, Python or related language

• Experience with neural deep learning methods and machine learning

• Experience with…