Store AI Agent - Onshore Data Engineer; Python, Pytorch, NodeJS

Contract (9 months 5 days)

Published 3 days ago

Databricks Platform

data modeling

mlops

Dev Ops & CI/CD

PySpark/Python

We are seeking a highly skilled Data Engineer with strong expertise in Databricks, Python, and PyTorch to design, develop, and optimize scalable data and machine learning pipelines. The ideal candidate will be responsible for building cloud‑native data platforms, enabling advanced analytics, and supporting AI/ML initiatives through robust data engineering practices.

This role requires hands‑on experience in large‑scale data processing, feature engineering, model deployment support, and performance optimization across modern data ecosystems.

Design, develop, and maintain scalable data pipelines using Databricks, PySpark, and Python.

Build and optimize batch and real‑time data ingestion frameworks from diverse data sources.

Develop ETL/ELT solutions to support enterprise data warehousing, analytics, and AI/ML workloads.

Implement data quality checks, validation frameworks, monitoring, and observability solutions.

Optimize Databricks jobs, notebooks, clusters, and workflows for performance and cost efficiency.

Design and maintain Delta Lake architectures, data models, and metadata management solutions.

Collaborate with Data Scientists and ML Engineers to prepare training datasets and feature stores.

Support development and deployment of deep learning solutions using PyTorch.

Develop scalable feature engineering pipelines and data transformation frameworks.

Enable model training, experimentation, and inference infrastructure on Databricks.

Integrate ML workflows with MLOps pipelines and CI/CD processes.

Implement Data Ops and MLOps best practices.

Work with cloud‑native services for storage, security, monitoring, and orchestration.

Support infrastructure‑as‑code and environment automation initiatives.

Partner with business, analytics, and data science teams to understand requirements and translate them into technical solutions.

Ensure compliance with data governance, privacy, and security standards.

Participate in architecture reviews and technology evaluations.

Strong experience with Databricks Platform

Expertise in Python

Strong hands‑on experience with Py Spark

Experience with PyTorch for machine learning and deep learning workloads

Knowledge of Delta Lake, Unity Catalog, and Databricks Workflows

Experience with ETL/ELT pipeline development

Strong SQL and data modeling skills

Experience with distributed computing and big data architectures

Knowledge of REST APIs and data integration patterns

Git version control and CI/CD tools

Experience with one or more:

AWS

Data Technologies:

Spark

Kafka/Event Streaming

Azure Data Factory or equivalent integration tools

Data Lake architectures

Data Warehousing solutions

Experience with GenAI and LLM data pipelines.

Knowledge of MLflow and MLOps frameworks.

Hands‑on experience with Docker and Kubernetes.

Experience with real‑time streaming architectures.

Familiarity with vector databases and AI platforms.

Certifications in Databricks or Cloud platforms.

Bachelor's or Master's degree in Computer Science, Data Engineering, Information Systems, or related field.

5+ years of experience in Data Engineering.

3+ years of experience with Databricks and cloud‑based data platforms.

Experience supporting machine learning and AI use cases using PyTorch.

Retail, QSR, Restaurant Technology, Digital Commerce, or Customer Analytics domains.

AI/ML and advanced analytics implementations.

The pay range that the employer in good faith reasonably expects to pay for this position is $39.30/hour - $61.40/hour. Our benefits…