Research Scientist

Research Scientist

Location:

Seattle, WA (Remote) Duration: 12 months contract with possible of extension

This role is for a Research Scientist who will be responsible for designing and implementing robust processes to evaluate and improve LLM-based models. More specifically, they will work on ensuring that we build models that will accelerate the developer work cycle and improve how software is built. This researcher will:

• Work closely with engineers, PMs, designers, and data scientists to align on research goals and build scalable end-to-end evaluation pipelines.
• Report on their research and make recommendations to stakeholders.
• Work autonomously and mentor other researchers to support the goals of the research organization.

Minimum Qualifications:

• 4+ years of experience in an applied research setting, or similar.
• Proven experience developing benchmarks and metrics to evaluate AI models, preferably with a focus on code generation or natural language processing (NLP).
• Familiarity with software engineering principles (unit tests, CI/CD, version control etc.).
• Strong programming skills in 1 or more programming languages (including Python).
• 2 years or more of experience with machine learning and data science algorithms and tools (e.g., PyTorch).
• Ability to produce meaningful visualizations and reports, and effectively present results to different audiences and across multiple media formats.
• Ability to work independently as well as collaboratively with minimal direction.
• Organized, highly attentive to detail, and manages time well.
• Experience working in industry.

Preferred Qualifications:

• Advanced degree (MS/PhD) in data analytics, statistics, or any engineering or computer science field.
• Experience with large language models (LLMs) and knowledge of state-of-the-art code generation techniques.
• Publication(s) in the fields of artificial intelligence, machine learning or data science.
• Familiarity with code generation-specific benchmarks and metrics.
• Ability to modify or fine-tune pre-trained models for specialized tasks.
• Familiarity with code scanning tools.