Data Engineer

Responsibilities

• Build scalable PySpark/Spark SQL pipelines in Databricks to prepare curated, feature ready datasets for model development and scoring.
• Develop and manage data ingestion and harmonization pipelines.
• Implement transformations, validations, and schema alignment based on model requirements.
• Optimize pipeline performance, scheduling, and orchestration (Jobs, Workflows, Delta Live Tables).
• Collaborate with Solution Architect, data architects and scientists to ensure accurate
• Integrate and harmonize data from Snowflake into Databricks, ensuring schema alignment and data consistency.
• Monitor pipeline health, data freshness, and quality, escalating issues early.
• Follow best practices for versioning, CI/CD, and operational MLOps workflows.
• Work closely with Data Scientists to translate feature logic into efficient, reusable pipelines.
• Implement orchestration and scheduling using Databricks Jobs/Workflows or Delta Live Tables.
• Monitor, optimize, and troubleshoot pipelines for performance, data quality, and SLAs.
• Ensure secure access patterns and compliance with customer data governance requirements.
• Support Dev/QA/Prod deployment of feature pipelines.

• 4–7 years of hands‑on experience in data engineering with cloud data platforms.
• Strong experience with Databricks, PySpark, Spark SQL, and Delta Lake.
• Proficiency with Snowflake and cross‑platform data integration.
• Experience building large‑scale feature engineering pipelines for ML workloads.
• Strong SQL skills and experience with distributed data processing.
• Understanding of MLOps concepts, pipeline versioning, and CI/CD best practices
• Strong understanding of Azure cloud platform, IAM, and enterprise security practices.