Principal, Data Scientist

Position: (USA) Principal, Data Scientist
Position Summary...

The Reliability Engineering group at Walmart Global Tech builds intelligent, data-driven platforms that ensure the availability, performance, and efficiency of Walmartʼs enterprise and e-commerce systems at massive scale. The team leverages large-scale telemetry, automation, and machine learning to enable

proactive optimization, faster incident detection, and resilient system behavior across thousands of services.

About the Team:

Building the right technology foundation for Infrastructure & Platforms is critical to operating at Walmartʼs scale. Our team designs and maintains the core technologies that power the broader tech organization - including data platforms, observability systems, Dev Ops tooling, cloud infrastructure, and runtime automation frameworks. These systems support secure, reliable, and scalable operations across stores, digital platforms, and distribution centers worldwide.

What you'll do...

As a Principle ML Engineer, you will architect, build, and operate production-grade ML systems that directly influence runtime behavior across large-scale distributed systems. This is a hands-on engineering role with strong system design and ownership responsibilities. You will:

* Architect and implement end-to-end ML systems (data pipelines, feature engineering, model training, deployment, and monitoring).

* Design scalable, low-latency model serving infrastructure integrated with Kubernetes and cloud- native systems.

* Build intelligent automation solutions including predictive autoscaling, anomaly detection, seasonality-aware forecasting, and capacity optimization.

* Engineer safe and reliable ML-driven automation that operates in high-availability environments.

* Own model lifecycle management, including validation, experiment tracking, model registry, monitoring, drift detection, and rollback strategies.

* Collaborate closely with platform, SRE, and infrastructure teams to embed ML capabilities into production systems.

* Drive engineering best practices around ML system reliability, observability, testing, and performance.

* Contribute to architectural decisions and mentor engineers on ML systems design.

Your solutions will operate at enterprise scale and directly impact system reliability, performance, and infrastructure cost efficiency. What Youʼll Bring:
Core Experience

* 10+ years of experience in software engineering with applied machine learning.

* Strong track record of building and operating ML systems in production.

* Experience owning systems end-to-end in distributed, high-availability environments.

* Experience leading technical initiatives or driving architectural decisions.

Technical Skills

* Strong proficiency in one or more programming languages commonly used in ML engineering, such as Python, Go, or Java.

* Strong experience with ML frameworks such as Scikit-learn, PyTorch, Tensor Flow, or similar.

* Strong SQL skills and experience working with large-scale datasets.

* Hands-on experience training, validating, and deploying machine learning models in production across domains such as recommendation systems, forecasting, anomaly detection, classification, or similar applied ML use cases.

* Experience building and maintaining end-to-end ML pipelines (data ingestion, feature engineering, training, evaluation, deployment, monitoring).

* Experience with model serving architectures (REST/gRPC inference services, batch inference, streaming inference).

* Hands-on experience with ML lifecycle platforms such as MLflow, Ray, Kubeflow, Airflow, or similar orchestration systems.

* Experience with experiment tracking, model registry, CI/CD for ML, feature management, and automated retraining workflows.

* Experience designing robust evaluation frameworks for traditional ML systems (offline validation, backtesting, shadow testing, A/B testing, and production performance monitoring).

* Strong experience working with observability data (metrics, logs, traces) and time-series analysis in distributed systems.

* Hands-on experience deploying and operating ML systems on Kubernetes, including containerization using Docker.

* Experience working with major cloud platforms (AWS, GCP, or Azure) and cloud-native services.

Systems & Architecture Expertise

* Strong understanding of distributed systems behavior (latency, throughput, failure modes, cascading effects).

* Ability to design ML systems that balance accuracy, latency, reliability, and safety.

* Experience designing fault-tolerant, observable, and scalable ML-driven automation systems.

* Solid understanding of cloud-native architecture and infrastructure patterns.

Nice to Have

* Experience building predictive or adaptive autoscaling systems.

* Experience with AIOps, demand prediction, anomaly detection, or incident prediction in distributed environments.

* Familiarity with streaming M L or online learning systems. Experience developing or integrating agentic systems, including orchestration, tool use, evaluation, and safety considerations in production environments.…