More jobs:
SPARK Developer-Remote
Remote / Online - Candidates ideally in
Washington, District of Columbia, 20001, USA
Listed on 2026-07-02
Washington, District of Columbia, 20001, USA
Listing for:
Staffing
Remote/Work from Home
position Listed on 2026-07-02
Job specializations:
-
Software Development
Data Engineering
Job Description & How to Apply Below
Spark Developer / Engineer
Workflows are powered by offline batch jobs written in Scalding, a Map Reduce-based framework. To enhance scalability and performance, migrating these jobs from Scalding to Apache Spark.
Key Responsibilities:- Understanding the Existing Scalding Codebase
- Analyze the current Scalding-based data pipelines.
- Document existing business logic and transformations.
- Migrating the Logic to Spark
- Convert existing Scalding jobs into Spark (PySpark/Scala) while ensuring optimized performance.
- Refactor data transformations and aggregations in Spark.
- Optimize Spark jobs for efficiency and scalability.
- Ensuring Data Parity & Validation
- Develop data parity tests to compare outputs between Scalding and Spark implementations.
- Identify and resolve any discrepancies between the two versions.
- Work with stakeholders to validate correctness.
- Writing Unit Tests & Improving Code Quality
- Implement robust unit and integration tests for Spark jobs.
- Ensure code meets engineering best practices (modular, reusable, and well-documented).
- Experience in big data processing with Apache Spark (PySpark or Scala).
- Strong experience with data migration from legacy systems to Spark.
- Proficiency in Scalding and Map Reduce frameworks.
- Experience with Hadoop, Hive, and distributed data processing.
- Hands-on experience in writing unit tests for Spark pipelines.
- Strong SQL and data validation experience.
- Proficiency in Python, Scala
- Knowledge of CI/CD pipelines for data jobs.
- Familiarity with Apache Airflow orchestration tool.
To View & Apply for jobs on this site that accept applications from your location or country, tap the button below to make a Search.
(If this job is in fact in your jurisdiction, then you may be using a Proxy or VPN to access this site, and to progress further, you should change your connectivity to another mobile device or PC).
(If this job is in fact in your jurisdiction, then you may be using a Proxy or VPN to access this site, and to progress further, you should change your connectivity to another mobile device or PC).
Search for further Jobs Here:
×