ML Researcher – Benchmarks & Evaluation

Deccan AI is a GenAI data company that runs the post-training and production layer for the world's leading AI labs and enterprises — from expert feedback and RL environments to managed AI work forces and evaluation infrastructure. Backed by A91 Partners, SIG (Susquehanna International Group), and Prosus Ventures, and having grown 10x last year, Deccan works with a majority of the Magnificent 7 and the top frontier labs globally.

Its global network of 1 million+ domain experts powers Helix, its hybrid eval suite, and Enterprise OS, its scaled ops automation platform. Headquartered in Mountain View, CA.

Job Description:

The ML Researcher plays a critical role in keeping the organization at the forefront of AI innovation through deep, end-to-end research. This role is focused on identifying emerging research directions, designing novel AI benchmarks, and creating high-quality evaluation datasets aligned with both cutting-edge research and organizational priorities.

The ML Researcher bridges the gap between theoretical research and practical application by translating insights from the latest AI papers into actionable benchmarks and evaluation frameworks. The role emphasizes deep research, synthesis of academic and industry findings, and the design of solutions that meaningfully advance AI systems—particularly in agent frameworks, machine learning models, and emerging AI paradigms.

The primary objective of the ML Researcher is to conduct comprehensive AI research and design novel, scalable, and real-world–relevant benchmarks and evaluation datasets
. The role aims to push beyond existing evaluation practices by defining new criteria that better measure the capabilities and limitations of modern AI systems.

Continuously track the latest AI research, conferences, and publications.

Conduct in-depth literature reviews across emerging AI models, agent frameworks, and evaluation methodologies.

Identify gaps, limitations, and opportunities in existing benchmarks.

Assess current industry benchmarks for relevance to organizational and client goals.

Propose novel AI benchmarks informed by research insights and industry gaps.

Design evaluation datasets that assess AI systems across both coding and non-coding domains.

Ensure benchmarks and datasets are innovative, scalable, and practically applicable.

Define meaningful evaluation metrics that provide actionable insights into model performance.

Create high-level requirement documents for each benchmark or dataset, covering:

Problem statement and motivation

Design overview and structure

Evaluation metrics and success criteria

Testing and validation guidelines

Ensure documentation is clear, comprehensive, and implementation-ready.

Work closely with the ML Lead, pipeline-focused MLEs, and project managers to align research outputs with organizational priorities.

Collaborate with internal stakeholders to ensure benchmarks meet project goals and industry standards.

Provide feedback and iterative improvements based on implementation outcomes.

Refine and evolve benchmarks and datasets based on feedback, new research, and emerging use cases.

Propose enhancements to existing evaluation methods, including new metrics or benchmark variations.

Stay actively engaged with the AI research community through conferences, discussions, and ongoing learning.

The ML Researcher is expected to produce high-impact, actionable research outputs, including:

Each proposal should include:

Clear problem definition and motivation

Detailed benchmark design and structure

Defined evaluation metrics

Testing and validation guidelines

Explanation of novelty…