Senior Data Engineer

• Design, develop, and maintain scalable data processing pipelines and workflows using frameworks such as Apache Spark, PySpark, and Apache Beam.
• Build and maintain microservices in Python that serve data-driven features in production.
• Develop internal tools to support CI/CD pipelines, experiment tracking, and data versioning.
• Collect, process, and integrate large datasets from multiple sources, including databases, file systems, and APIs.
• Ensure data integrity, consistency, and quality through robust validation and monitoring processes.
• Optimize data systems for performance, scalability, and high availability.
• Implement best practices for data security, access control, and privacy.
• Collaborate with data scientists, analysts, and engineers to support analytics and ML workflows.
• Lead complex migration initiatives involving the transition from on-premise Cloudera environments to cloud-native platforms, ensuring zero data loss and minimal downtime.

• Strong understanding of distributed systems and modern data architectures.
• Proven, hands‑on experience with large-scale data platform migrations, specifically transitioning from Cloudera (CDH/HDP) to either Snowflake or Databricks.
• Deep technical expertise in building and orchestrating a high‑performance data platform from scratch.
• 5+ years of professional experience in software engineering or data engineering.
• Strong software engineering skills with Python in large‑scale, high‑performance production environments.
• Hands‑on experience with Spark/PySpark and other big data frameworks.

Demonstrates expertise in designing and maintaining scalable data processing pipelines using frameworks like Apache Spark and PySpark, with a strong focus on data integrity and performance optimization. Proven ability to lead data platform migrations and collaborate effectively with cross‑functional teams to support analytics and machine learning workflows.

• Apache Spark
• Py Spark
• Data Platform Migration
• Python Software Engineering
• Distributed Systems

• Data Processing Pipelines
• Microservices Development
• CI/CD Pipelines
• Data Integration
• Data Validation
• Performance Optimization
• Data Security
• Access Control
• Cloud‑Native Platforms
• Big Data Frameworks

• Collaboration
• Problem‑Solving
• Communication

• Data Architecture
• Data Quality
• Data Versioning
• Analytics Workflows
• Machine Learning

• Apache Beam
• Cloudera
• Snowflake
• Databricks