Computational Linguist, AI Evaluation

About the Role:

You will be a vital member of our ML Data Operations Team – which leads the full spectrum of video-language data collection, labeling operations, and model quality measurement. This role comes with high ownership and includes responsibilities such as defining dataset and evaluation requirements in consultation with our research and product teams; designing and building data pipelines; and coordinating with our vendor partners that execute  will also be responsible for automating as much of the repetitive partnership and annotation-quality-evaluation work as possible.

A desire to work cross functionally and to build relationships is critical for success in this position. Position is hybrid in San Francisco - onsite Tuesdays & Thursdays.

• Evaluation Strategy & Design:
  Define and build evaluation protocols for video-language model quality, working with Research and Product teams to translate ambiguous quality questions into measurable, repeatable benchmarks or evaluation flows.

• Data-to-Insight Pipelines:
  Turn raw customer and usage data into structured signal, building pipelines and analyses that surface where our models are underperforming and working with XFN partners to translate these insights into concrete improvement plans.

• Labeling Operations:
  Design and execute video-language data collection and labeling projects, automating repetitive processes so the team can focus on higher-leverage work.

• Vendor & Partner
  
  Collaboration:
  
  Coordinate with vendor and outsourcing partners executing at scale, keeping quality high through clear guidelines and feedback loops.

• Cross-Functional Partnership:
  Work closely with Engineering and Research teams to align on top-priority data and evaluation needs, communicating findings through dashboards and reports that drive decisions.

• Direct experience building or running model evaluation pipelines (benchmark design, human eval frameworks, automated scoring systems), and translating ambiguous quality questions into measurable criteria.

• 5+ years of experience working in an AI focused data operations organization.

• A proven track record designing and executing large scale data projects, including gathering, labeling, and post-processing data.

• The ability to analyze messy and complex data, identify overarching patterns, and distill your findings into crisp annotation guidelines or other accessible documentation.

• Proficiency with Python, agentic coding, or other popular industry tools for automation.

• Excellent communication and project management skills, and the ability to support several projects simultaneously.

• A foundational understanding of and interest in LLMs/VLMs and multimodal AI.

• Conviction that data is the key ingredient for the performance of AI models.

• Experience in data collection and labeling for multimodal language models.

• Experience working with research scientists and engineers.

• Experience planning and operating new data tools and labeling systems.

• Expertise or interest in video-centric domains, such as sports, advertising, and content creation.

• Technical:
  Python, FFMPEG, AWS (S3, EC2)

• Project Management:
  Linear, Notion, Google Suite, Encord

Even if there are a few check boxes that aren't ticked through your prior experience, we still encourage you to apply! If you are a 0-1 achiever, a ferocious learner, and a kind and fun team player who motivates others, you will find a home at Twelve Labs.

We are a global company that values the uniqueness of each person's journey. It is the differences in our cultural, educational, and life experiences that allow us to constantly challenge the status quo. We are looking for individuals who are motivated by our mission and eager to make an impact as we push the bounds of technology to transform the world.

Join us as we revolutionize video understanding and multimodal AI.

An open and inclusive culture and work environment

Work closely with a collaborative, mission-driven team on cutting-edge AI technology

Full health, dental, and vision benefits

Extremely flexible PTO and parental leave policy. Office closed the week of Christmas and New Years

Monthly wellness stipend

Annual Learning & Development stipend to invest in your growth

Global offices in San Francisco and Seoul, and co-working office memberships for remote team members

Transportation stipend (SF onsite/hybrid only)

Daily lunch & dinner provided (SF onsite/hybrid only)