Senior Applied Scientist

** Job Description*
* The OCI AI Evaluation Science team builds the evidence behind model-selection, product-readiness, and launch decisions. We evaluate frontier foundation models and AI systems across capabilities such as reasoning, coding and agentic coding, retrieval-augmented generation, AI agents, NL2

SQL, multimodal understanding, multilingual performance, and responsible AI.

As a Senior Applied Scientist on the team, you will independently own complex evaluation work from problem definition, benchmark development, and final recommendation to executive leadership. You will translate ambiguous product and customer questions into measurable hypotheses, select or create appropriate benchmarks, design experiments, build evaluation pipelines, validate data and metrics, analyze failure modes, and communicate conclusions to science, engineering, product, and leadership stakeholders.

This is hands-on applied science. You will write high-quality code, work with large and imperfect datasets, develop and calibrate automated evaluators, and turn one-off analyses into reproducible evaluation protocols and reusable infrastructure. You will examine more than aggregate benchmark scores, considering factors such as statistical validity, data provenance, contamination, robustness, cost, latency, reliability, safety, and operational constraints.

The work sits at the point where research results become product decisions. Success requires scientific rigor, strong engineering judgment, clear writing, and the ability to make progress when requirements, model access, data, or infrastructure are still evolving. You will collaborate closely with other scientists, software engineers, product teams, data and human-annotation teams, and external partners to deliver evaluation results that are technically defensible and useful in practice.

You will develop novel benchmarks and evaluation methodologies that are publishable at top tier AI conferences.

** Responsibilities*
* ** Key Responsibilities*
* + Independently own end-to-end evaluations of foundation models, AI agents, and enterprise AI systems, from initial question and experiment design through analysis, reporting, and stakeholder review.

+ Translate customer, product, and business needs into testable hypotheses, evaluation criteria, datasets, metrics, baselines, and acceptance thresholds.

+ Design, implement, and maintain benchmarks and evaluation methods for areas such as but not limited to reasoning, coding, agentic workflows, RAG, NL2

SQL, multimodal systems, multilingual performance, and responsible AI.

+ Publish original research in top-tier peer-reviewed conferences and journals, and translate relevant evaluation advances into reusable methods, technical reports, or production capabilities for OCI.

+ Write production-quality evaluation code; build reproducible pipelines, test suites, automated checks, and integrations with shared evaluation platforms.

+ Evaluate model and system behavior across quality, cost, latency, reliability, safety, robustness, and domain fit rather than relying only on aggregate scores.

+ Conduct statistical analysis, error analysis, ablations, and qualitative failure-mode investigations to explain model behavior and identify meaningful differences between systems.

+ Develop and validate automated evaluators, including LLM-as-a-judge and VLM-as-a-judge methods; calibrate them against human judgments and quantify their reliability, bias, and limitations.

+ Design human-evaluation and annotation workflows, including rubrics, gold datasets, sampling plans, quality controls, and vendor or Human-in-the-Loop validation.

+ Assess dataset quality, provenance, representativeness, contamination risk, licensing constraints, privacy, and other factors that could invalidate an evaluation or limit use of its results.

+ Produce concise, decision-ready reports that make methods, assumptions, limitations, tradeoffs, and recommendations explicit for technical and non-technical audiences.

+ Partner with science, engineering, product, data, and operations teams to define requirements, resolve blockers, manage dependencies, and establish clear handoffs and ownership.

+ Turn successful evaluation work into reusable protocols, documented workflows, and shared infrastructure that improve the speed and consistency of future evaluations.

+ Stay current with research in machine learning, generative AI, agent evaluation, and measurement methodology; prototype promising approaches and contribute to science plans, papers, patents, or technical reports where appropriate.

+ Review technical work, share expertise, and mentor junior scientists or engineers in experimental design, evaluation methodology, coding, and interpretation of results.

+ Own delivery quality and timelines, communicate risks early, and maintain clear, auditable documentation of experimental configurations, data versions, results, and decisions.

Disclaimer:

** Certain U.S. based or U.S. customer or client-facing roles may be…