Skip to content
×

Data Engineer; Java​/Spark​/Hadoop

Job in Boise, Ada County, Idaho, 83701, USA
Listing for: Distillery
Full Time position Listed on 2026-11-09
Job specializations:
Salary/Wage Range or Industry Benchmark: 120000 - 180000 USD Yearly USD 120000.00 180000.00 YEAR

Job Description & How to Apply Below

Position: Data Engineer (Java/Spark/Hadoop)

Distillery is a software development partner that helps organizations design, build, and scale modern technology solutions. Leading companies work with us to accelerate digital initiatives, solve complex technical challenges, and bring innovative products to market faster. Working as an extension of our clients' teams, we deliver lasting business value. Distillery is committed to diversity and inclusion. We embrace a dynamic and inclusive culture where you have the opportunity to continuously learn and grow.

About

the Position

We are seeking a hands-on Data Engineer to join the data engineering team of a US-based technology client as the dedicated engineer for a set of platform migrations. In this role, you will move Java and Spark pipelines from Cloudera to AWS EMR, upgrade the remaining Cloudera workloads to the latest CDP version, and migrate orchestration from Airflow 2 to Airflow
3. Working with a small, experienced team that uses AI-assisted development tools day to day, you will execute the technical direction set by the client's staff engineers, making sure each pipeline is migrated, tested, and running reliably in production. This is a full-time, long-term position with immediate start, requiring at least 5 hours of overlap with US Pacific Time.

Responsibilities
  • Migrate Spark batch and Map Reduce pipelines written in Java from Cloudera to AWS EMR, adapting code, configuration, and business logic as needed.
  • Upgrade HBase-dependent pipelines to the newest Cloudera CDP version, including Spark and Java version upgrades.
  • Validate each migrated pipeline end to end, covering data parity, performance, and failure handling.
  • Update Airflow DAGs and support the migration of orchestration from Airflow 2 to Airflow 3, ideally in the same pass as each pipeline migration.
  • Support a small number of streaming pipelines through the same migrations.
  • Build new ETL pipelines based on stakeholder requests.
  • Troubleshoot and resolve production issues in existing pipelines.
  • Document migration steps and findings so the rest of the team can reuse them.
Requirements
  • 4+ years of experience in data engineering or backend software engineering, building and running data pipelines in production.
  • Strong Java skills, with solid backend engineering fundamentals (build tools, dependency management, testing, and debugging).
  • Hands-on experience with Apache Spark, ideally Spark batch jobs written in Java.
  • Working knowledge of the Hadoop ecosystem: HDFS, YARN, Map Reduce, Hive, or HBase.
  • Experience scheduling and maintaining pipelines with Apache Airflow.
  • Experience upgrading or migrating pipelines between platforms or framework versions, and validating that results remain unchanged.
  • Ability to work independently, raise risks early, and keep work moving without close supervision.
  • Professional English, both spoken and written.
Nice to Have
  • Experience with AWS EMR, S3, and related AWS data services.
  • Cloudera CDH or CDP administration or upgrade experience.
  • Experience migrating from Airflow 2 to Airflow 3.
  • Experience with Spark Structured Streaming or other streaming frameworks.
  • Familiarity with DynamoDB, Redis, or other alternatives to HBase.
  • Day-to-day use of AI coding tools such as Cursor or Claude.
To View & Apply for jobs on this site that accept applications from your location or country, tap the button below to make a Search.
(If this job is in fact in your jurisdiction, then you may be using a Proxy or VPN to access this site, and to progress further, you should change your connectivity to another mobile device or PC).
View / Apply for Jobs
Matching My Jurisdiction

Search for further Jobs Here:

SEARCH JOBS: 10+ Million Listings
(Try combinations for better Results! Or enter less keywords for broader Results)
Education Level
Experience Level (years)
Posted in last:
Salary