Senior Spark Data Engineer - Databricks & ML Pipelines
Listed on 2026-05-28
-
Software Development
Data Engineer
We are seeking a Senior Data Engineer with deep expertise in Apache Spark and hands‑on experience delivering production‑grade solutions on Databricks. This role centers on designing, optimizing, and operating large‑scale Spark data platforms that power analytics, machine learning (ML), and artificial intelligence (AI) use cases.
The ideal candidate is a Spark practitioner who understands how to build efficient distributed data pipelines, optimize performance and cost, and enable ML workflows will lead Spark pipeline design, collaborate closely with data scientists and analysts, and help operationalize ML and AI solutions across the enterprise.
Key Responsibilities
- Design, build, and maintain distributed data pipelines using Apache Spark on Databricks
- Lead development of high‑performance Spark jobs for batch and near‑real‑time processing
- Implement complex data transformations using PySpark (Data Frame & Spark SQL APIs)
- Apply Spark best practices for partitioning, caching, joins, and shuffles
- Tune Spark jobs for performance, scalability, and cost efficiency
Required Qualifications
- 12+ years of experience in Data Engineering.
- Strong hands‑on experience building and operating Apache Spark applications.
- Proven experience running Spark workloads in Databricks (production environments).
- Advanced proficiency in PySpark, Spark SQL, and Python.
- Experience working with large‑scale, distributed datasets in cloud environments (AWS preferred).
- Demonstrated experience supporting ML or AI pipelines on Spark.
- Strong understanding of data modeling, distributed systems, and data architecture.
- Experience designing reliable, automated, and observable Spark pipelines.
About Value Momentum
Value Momentum is a product development company. We have 25+ years in the market and we have products like Ifoundry, rating engine and Biz dynamics. We have been partnered with Guidewire, Duck Creek and Insurity . We have headquartered in NJ and PA. we have specialised in P&C insurance, we have 75+ clients implementing and delivering. we do have offshore development centres India, Canada, and UK.
we have overall 6000+ employees overall the globe.
(If this job is in fact in your jurisdiction, then you may be using a Proxy or VPN to access this site, and to progress further, you should change your connectivity to another mobile device or PC).