Sr Data Scientist
Job in
Santa Clara, Santa Clara County, California, 95053, USA
Listing for:
Amazon Development Center U.S., Inc.
Full Time
position
Listed on 2026-11-15
Job specializations:
-
IT/Tech
Job Description & How to Apply Below
This role focuses on leading the evaluation and benchmarking data development that supports Amazon Quick. You will design datasets, define metrics and evaluation frameworks, and use the resulting insights to guide product decisions and model improvements.
Responsibilities- Design and build evaluation and benchmark datasets.
- Define quality metrics and evaluation frameworks, including human evaluation, model-based evaluation using LLM-as-a-Judge
, and checks on the correlation between human and model evaluations. - Run human annotation programs end to end, including writing guidelines, measuring inter-annotator agreement, and managing data quality controls.
- Analyze model and agent performance to identify gaps, root causes, and the most impactful improvement opportunities.
- Collaborate with science, engineering, and product teams to translate evaluation results into launch decisions and roadmap priorities.
- Present findings and recommendations to senior leadership.
- 5+ years of experience with data querying languages (for example,
SQL
), scripting languages (for example,
Python
), or statistical and mathematical software (for example, R,
SAS
, Matlab
). - 4+ years of data scientist experience.
- Experience leading and mentoring on a data science team.
- 2+ years of experience in data visualization using AWS Quick Sight
, Tableau
, R Shiny
, or similar tools. - Experience managing data pipelines.
- Experience evaluating LLMs
, RAG systems, or agentic AI applications.
SQL
, Python
, R,
SAS
, Matlab
, AWS Quick Sight
, Tableau
, R Shiny
, LLM-as-a-Judge
, RAG
Location: Santa Clara, CA (onsite).
Salary: USD 183, per year.
- Comprehensive benefits including health insurance (medical, dental, vision, prescription, Basic Life & AD&D insurance and option for Supplemental life plans), EAP, mental health support, medical advice line, flexible spending accounts, and adoption and surrogacy reimbursement coverage.
- 401(k) matching.
- Paid time off.
- Parental leave.
- Sign-on payments and restricted stock units (RSUs).
To View & Apply for jobs on this site that accept applications from your location or country, tap the button below to make a Search.
(If this job is in fact in your jurisdiction, then you may be using a Proxy or VPN to access this site, and to progress further, you should change your connectivity to another mobile device or PC).
View / Apply for Jobs
Matching My Jurisdiction
(If this job is in fact in your jurisdiction, then you may be using a Proxy or VPN to access this site, and to progress further, you should change your connectivity to another mobile device or PC).
View / Apply for Jobs
Matching My Jurisdiction