Principal Data Scientist

Lead data science and AI evaluation at Staff or Principal level, making performance measurable, reproducible, and useful in real operational workflows.

Turn ambiguous quality questions into evidence that guides product decisions. Partner with engineers and domain experts to evaluate model behavior, improve retrieval and prediction, and understand the cost of errors in sensitive or consequential work.

This opening considers candidates at Staff or Principal level. Staff Data Scientists lead evaluation strategy and substantial applied research across product and engineering teams, establish rigorous methods, and mentor colleagues. Principal Data Scientists shape scientific direction across products, guide high-impact methodological decisions, and establish standards for evidence and model evaluation. Experience minimums apply to the level under consideration; demonstrated scientific leadership, scope, and impact also determine level.

• Design representative evaluation datasets, labeling guidance, and repeatable experiments for AI-assisted workflows.
• Measure factual quality, retrieval relevance, task completion, uncertainty, and performance across important user and data segments.
• Compare models and system configurations using quality, latency, inference cost, and human review effort.
• Investigate failure patterns and translate findings into product changes, better datasets, and release criteria.
• Document data provenance, evaluation limitations, and reproducibility while protecting sensitive information.

• Staff level: at least 8 years of relevant professional experience in applied data science, machine learning, or quantitative research.
• Principal level: at least 12 years of relevant professional experience in applied data science, machine learning, or quantitative research.
• Prior professional experience working at an AI-forward company is required.
• A bachelor’s degree or higher in Computer Science, Data Science, Statistics, Mathematics, Artificial Intelligence, Machine Learning, or a closely related quantitative field from an accredited or officially recognized institution.
• A record of independently leading significant AI or ML work, setting evaluation standards, and guiding technical decisions with clear evidence.
• Strong programming skills in Python and Go are required, alongside SQL proficiency, practical statistics, experimental design, and experience working with imperfect real-world datasets.
• Experience evaluating machine learning systems and interpreting uncertainty, bias, and variation in results.
• Strong applied understanding of LLMs, embeddings, retrieval, and the limits of model-based evaluation.
• Ability to communicate findings and tradeoffs clearly to engineering, product, and domain stakeholders.

• Experience with retrieval evaluation, human labeling programs, drift monitoring, or production model monitoring.
• Work with privacy-sensitive data or operational decisions that require defensible evidence.
• A master’s degree or PhD in a relevant technical or quantitative discipline.

This is a full-time position based in our Burlingame, California office and requires working on-site five days a week.

Robust theoretical and applied knowledge of AI and machine learning, including supervised and unsupervised learning, deep learning, generalization, calibration, sampling bias, and leakage-resistant validation.

Every team member should be able to discuss the capabilities and limitations of AI relevant to their work, verify AI-generated output, protect sensitive information, and keep their knowledge current as the technology evolves.

We work directly with individual applicants. We do not work with recruiting agencies, consulting firms, offshore or outsourcing companies, or freelancers offering external services for these openings. Please do not submit candidates or service proposals on behalf of a third party. Individuals may apply directly for our full-time positions.

All applicants must be legally authorized to work in the United States without employer sponsorship. Metrosion does not provide employment visa sponsorship.

All positions require working on-site in our Burlingame, California office five days a week. These roles are not remote or hybrid.

We welcome applicants from all backgrounds and value an inclusive, respectful workplace.