Tech Lead Machine Learning Engineer - LLM Evaluation & Agent Systems

Tik Tok USDS JV is looking for a Senior Machine Learning Engineer technical leader to help build an AI Risk & Compliance Intelligence Platform and an Intelligent Data Agent Ecosystem. This onsite role in Seattle, WA combines architecture and execution across LLM evaluation, synthetic data generation, and modular agent systems that support Data, Engineering, and SRE workflows.

You will work within the Tik Tok U.S. Data Security (USDS) team, which focuses on the security, integrity, and compliance of Tik Tok's data in the United States. In parallel, collaboration with the Data Platform Team will support scalable, reliable, and cost-efficient infrastructure for self-service tools and scientific insights.

• Design, build, and scale robust LLM evaluation architecture
  , including automated offline datasets, metric suites, and benchmarking pipelines to assess prompt and model performance.
• Develop LLM-driven synthetic data generation approaches to address data scarcity bottlenecks and enable systematic evaluation of long-tail and edge-case risk scenarios.
• Move beyond criteria-based prompt engineering by extracting risk patterns from complex case data to create native, ML-driven decision models and reasoning abstractions.
• Architect, evaluate, and continuously optimize modular agent skills and tools tailored for Data Science, Analytics, Data Engineering, and SRE workflows, improving tool-use accuracy, execution efficiency, and task success rates.
• Apply advanced evaluation methodologies (including LLM-as-a-judge and multi-agent benchmarking) across both risk intelligence and agentic ecosystems.
• Provide technical leadership by defining core ML system design patterns, benchmarking standards, and engineering best practices, and by translating complex operational requirements into scalable ML solutions with cross-functional partners.

• 5+ years of software or ML engineering experience, including a strong track record designing, building, and deploying ML/LLM systems in production.
• Hands‑on expertise with LLM architectures
  , fine‑tuning, RAG, prompt tuning, and agentic frameworks.
• Experience building robust LLM evaluation frameworks (LLM-as-a-judge, automated benchmark pipelines) and working on data scarcity or imbalance problems.
• Strong coding skills with production experience supporting scalable data processing pipelines.
• Prior domain experience in Risk & Compliance Intelligence and Automated Decision Systems.
• Hands‑on experience with synthetic dataset generation or bootstrapping models for cold‑start scenarios.
• Experience building AI tools or agents for technical personas including Data Analysts, Data Scientists, Data Engineers, and Data SRE.
• Demonstrated ability to serve as a founding technical lead on ambiguous, high-impact initiative pods.

• Location: Seattle, WA (onsite)
• Salary: USD 198, per year

• Day one access to medical, dental, and vision insurance
• 401(k) savings plan with company match
• Paid parental leave
• Short-term and long-term disability coverage
• Life insurance
• Wellbeing benefits, among others
• 10 paid holidays per year
• 10 paid sick days per year
• 17 days of Paid Personal Time (prorated upon hire with increasing accruals by tenure)

Tik Tok USDS Joint Venture LLC is dedicated to the safety and security of millions of Americans who create, discover, and connect with what they love on the apps we operate. On-site presence across teams is intended to improve speed, alignment, and agility, particularly for real-time decision-making, team development, and integrated execution. The company is shifting from a hybrid work model to a fully in-person schedule up to 5 days a week.

USDS is committed to providing reasonable accommodations in its recruitment processes for candidates with disabilities, pregnancy, sincerely held religious beliefs, or other reasons protected by applicable laws.