Research Scientist, AI Evaluation Science

Research Scientist, AI Evaluation Science

Seattle, Washington, United States Software and Services

AI systems are only as trustworthy as the methods used to evaluate them. At Apple, where AI powers experiences for billions of people, getting evaluation right is not a support function—it is a foundational science. Our team, part of Apple Services Engineering, is building that scientific foundation: rigorous, scalable evaluation methodology for LLMs, agentic systems, and human‑AI interaction. What makes this team unusual is its interdisciplinary core.

You will work alongside measurement scientists (psychometrics, validity theory), ML researchers, and platform engineers—bringing together ML research, statistical rigor, and production engineering. We are looking for a Research Scientist who treats evaluation methodology itself as a first‑class research problem—someone with deep technical fluency in preference learning, reward modeling, or calibration theory, and the drive to advance the field while solving real problems 're hiring at multiple levels (early‑career to senior researchers).

What unites all candidates is depth of thinking about evaluation as a research problem.

This is primarily a research role. You will formulate open problems in evaluation science, design experiments, publish findings, and drive projects from conception through completion. While you will also partner with platform engineers to ensure your methods are productionized into SDKs and APIs, the focus of the role is original research. Our research team brings together ML scientists and measurement scientists to tackle evaluation as both a machine learning and a measurement problem, building methods that are technically innovative and scientifically valid.

You will also work closely with a platform engineering team that translates research into production‑ready SDKs and APIs used across Apple. The successful candidate will have a strong publication record in evaluation‑adjacent ML areas and a demonstrated ability to implement complex methods from recent papers, run large‑scale experiments, and communicate results to both technical and non‑technical audiences.

• Advance evaluation methodology through original research in one or more of the following areas: preference learning and reward modeling (RLHF, DPO, reward hacking mitigation); LLM‑as‑judge calibration, rubric design, and bias detection; intelligent evaluation strategies including active learning for test selection and automated failure discovery; or validity frameworks for evaluators (construct validity, transfer learning). You are not expected to cover all of these—depth matters more than breadth.
• Publish at top‑tier venues (NeurIPS, ICML, ICLR, ACL, EMNLP), contributing to evaluation science as a recognized research area and representing Apple in the research community.
• Translate research into production‑ready tools by partnering with platform engineers to product ionize your methods into evaluation SDKs and APIs used across Apple.
• Collaborate with measurement scientists to integrate psychometric methods and validity frameworks into evaluation systems, ensuring evaluators measure what they claim to measure.
• Define the team's research agenda for evaluation science by identifying high‑leverage open problems, validating that they address real‑world challenges faced by ML engineers across Apple, and designing rigorous experimental programs to solve them.

• Ph.D. in Computer Science, Machine Learning, or a closely related field, with a research focus in evaluation‑adjacent areas (preference learning, RLHF, human feedback, calibration, automated assessment)
• Strong publication record at top‑tier conferences (NeurIPS, ICML, ICLR, ACL, EMNLP), including first‑author publications demonstrating independent research contributions
• Deep technical expertise in at least one evaluation‑adjacent ML area, with strong mathematical foundations: preference learning and reward modeling (RLHF, DPO, reward hacking, specification gaming); OR calibration theory, proper scoring rules, and statistical reliability; OR human‑AI interaction methodology…