Software Development Engineer in Test; Machine Learning

Requirements

• Bachelor's degree or equivalent
• 3+ years of experience as an SDET, with strong experience testing backend systems
• Proficiency in Python/Java/Scala/go for test automation
• Hands-on experience with API testing tools (e.g., Postman or similar) for distributed systems
• Familiarity with CI/CD pipelines and test execution in Jenkins or similar environments such as jenkins, Git Hub Actions, ArgoCD, etc
• Hands-on experience building and maintaining automated test frameworks (e.g., pytest, JUnit, TestNG, etc.)Experience in working with databases for backend validation
• Experience with cloud and container technologies (AWS, GCP, Kubernetes, Docker, etc.)
• Familiarity with monitoring and observability tools (e.g., Prometheus, Grafana, Datadog, etc.) for validating service health
• Strong understanding of software development lifecycle (SDLC) and agile methodologies
• Excellent problem-solving skills and excellent communication skills
• (Desirable) Experience testing machine learning systems, such as recommendation systems, search ranking systems, or other probabilistic ML application systems
• (Desirable) Experience creating synthetic or seeded test data to simulate realistic customer/account behaviors
• (Desirable) Experience evaluating model outputs using statistical techniques, distribution analysis, or scenario-based validation, rather than deterministic assertions
• (Desirable) Knowledge of data pipelines, feature stores, inference systems or model-serving infrastructure (Seldon, KServe, Ray Serve, etc.) is a plus
• (Desirable) Experience testing online services with high RPS and low latency requirements
• (Desirable) Experience with performance and load testing tools (Locust, k6, Gatling, JMeter, etc.)
• (Desirable) Experience with Databricks or similar ML platform tooling

• Play Station is seeking a Software Development Engineer in Test, ML to join our San Mateo team and lead quality efforts for machine learning-powered products and services
• You will partner closely with ML engineers, software engineers, product managers, and data partners to build scalable test automation and validation approaches for systems serving Play Station at global scale
• This role is focused on building confidence in model reliability, production readiness, and behavioral consistency
• Unlike traditional software backend testing, success here requires validating probabilistic systems through score distributions, cohort-level behavior, thresholds, and statistical signals, not just deterministic pass/fail assertions
• Design, develop, and maintain automated test frameworks, test scripts for ML inference services and related ML platform components using python/Java
• Build and automate realistic, representative test account creations and datasets to evaluate model behavior across different scenarios
• Validate score distributions, ranking behavior, output quality, and other model signals for incoming ML models prior to release
• Perform different types of testing, including functional, integration, regression and API testing working with RESTful APIs, microservices and databases, and non-functional performance and load testing to verify inference service scalability, latency requirements and reliability under production like conditions
• Work closely with ML engineers and ops teams to ensure that all features and bug fixes come with automated test coverage, ensuring continuous integration and deployment
• Debug and analyze test failures, model anomalies and report defects
• Drive high standards of quality for both engineering decisions and customer-facing features with optimal test and automation strategy
• Develop and implement best practices for test automation that is highly scalable and maintainable
• Monitor test execution and optimize performance in test environments
• Come up with innovative solutions to organizational problems