Research Scientist (Model Evaluation

About the Role

Progress in speech AI is only as meaningful as our ability to measure it. At Sanas, model quality spans dimensions that automated metrics struggle to capture — accent naturalness, perceptual clarity, speaker identity preservation, noise suppression without speech distortion, translation fluency under real-world disfluency. We're looking for a Research Scientist who can define what "better" actually means across all of Sanas's model families, build the evaluation infrastructure to measure it rigorously, and close the loop between research progress and real-world impact.

This role sits at the intersection of research, product, and infrastructure — and directly shapes how every model team at Sanas measures progress.

• Design and own evaluation frameworks across Sanas's full model portfolio — Accent Translation, Noise Cancellation, Speech Enhancement, and Language Translation, and more — ensuring each captures meaningful progress, not just benchmark performance.
• Develop novel quantitative metrics for subjective and perceptual qualities: accent similarity, naturalness, speaker identity preservation, intelligibility under noise, and translation fluency in spoken-language domains.
• Build evaluation systems that bridge automated metrics and human judgment — designing listening studies, MOS/MUSHRA protocols, and preference tests that are statistically rigorous and operationally scalable.
• Define evaluation splits, test sets, and benchmark suites that accurately reflect production conditions — diverse accents, languages, noise environments, recording devices, and telephony codecs.

• Build and maintain automated evaluation pipelines that run continuously against model checkpoints — surfacing regressions early and tracking quality trends across training runs.
• Develop reference-based and reference-free metrics calibrated to Sanas's specific model tasks: SI-SDR, PESQ, STOI, DNSMOS, speaker similarity, WER delta, COMET, and task-specific custom metrics where off-the-shelf measures fall short.
• Instrument model quality monitoring in production — detecting degradation across language pairs, accent profiles, and acoustic conditions in live customer traffic.
• Build tooling that allows research scientists and ML engineers to run rigorous ablations, compare model versions, and understand quality tradeoffs without needing to design the evaluation from scratch each time.

• Design and operate human evaluation programs — listener panels, crowdsourced annotation, and expert evaluator workflows — that produce reliable signal on dimensions automated metrics cannot capture.
• Conduct research into evaluation methodology itself: when do automated metrics correlate with human perception, when do they diverge, and what does that tell us about model behavior?
• Partner directly with research scientists across model teams to translate open-ended quality questions into concrete, measurable evaluation protocols.

• Work closely with ML research, product, and customer success teams to ensure evaluation reflects what customers actually experience — not just what lab conditions optimize for.
• Feed evaluation insights back into data acquisition and model training priorities — identifying which failure modes require more data, architectural changes, or training procedure improvements.
• Communicate evaluation results clearly to both technical and non-technical stakeholders, translating metric movements into product quality narratives that inform roadmap decisions.

• 4+ years of research or applied research experience in speech, audio, or NLP, with a demonstrated focus on evaluation methodology and quality…