Senior Machine Learning Engineer - Model Evaluations, Public Sector

Senior Machine Learning Engineer - Model Evaluations, Public Sector

The Public Sector ML team at Scale deploys advanced AI systems—including LLMs, agentic models, and multimodal pipelines—into mission-critical government environments. We build evaluation frameworks that ensure these models operate reliably, safely, and effectively under real-world constraints. As an ML Engineer, you will design, implement, and scale automated evaluation pipelines that help customers trust and operationalize advanced AI systems across defense, intelligence, and federal missions.

• Develop and maintain automated evaluation pipelines for ML models across functional, performance, robustness, and safety metrics, including LLM-judge-based evaluations.
• Design test datasets and benchmarks to measure generalization, bias, explainability, and failure modes.
• Build evaluation frameworks for LLM agents, including infrastructure for scenario-based and environment-based testing.
• Conduct comparative analyses of model architectures, training procedures, and evaluation outcomes.
• Implement tools for continuous monitoring, regression testing, and quality assurance for ML systems.
• Design and execute stress tests and red-teaming workflows to uncover vulnerabilities and edge cases.
• Collaborate with operations teams and subject matter experts to produce high-quality evaluation datasets.
• Comfortable with light travel (approximately 10%) for customer interaction and team needs.

Active security clearance or the ability to obtain a security clearance is required for this role.

• Experience in computer vision, deep learning, reinforcement learning, or NLP in production settings.
• Strong programming skills in Python; experience with Tensor Flow or PyTorch.
• Background in algorithms, data structures, and object-oriented programming.
• Experience with LLM pipelines, simulation environments, or automated evaluation systems.
• Ability to convert research insights into measurable evaluation criteria.

• Graduate degree in CS, ML, or AI.
• Cloud experience (AWS, GCP) and model deployment experience.
• Experience with LLM evaluation, CV robustness, or RL validation.
• Knowledge of interpretability, adversarial robustness, or AI safety frameworks.
• Familiarity with ML evaluation frameworks and agentic model design.
• Experience in regulated, classified, or mission-critical ML domains.

EEO:
Scale is an equal opportunity employer committed to inclusion and diversity in the workplace.