Big Data Developer

Overview

We’re looking for a Big Data Engineer to play a key role in modernizing enterprise‑scale data platforms. This person will focus on migrating and optimizing legacy Hadoop‑based workloads into open‑source and cloud‑aligned environments. The ideal candidate combines strong technical skills across distributed systems with a passion for performance, scalability, and clean data architecture.

Big Data Developer

Remote USA

$130,000 - $150,000

Healthcare, PTO

• Rebuild and optimize Spark, Hive, and Map Reduce workloads for execution in modern distributed compute environments.
• Design, develop, and maintain data pipelines for both batch and streaming use cases using Spark, NiFi, and Kafka.
• Convert legacy ETL processes and workflows into open‑source or cloud‑native frameworks.
• Collaborate with data engineers, architects, and platform teams to define data ingestion and transformation standards.
• Tune and optimize performance across large‑scale data workloads (partitioning, caching, resource management).
• Implement and automate data validation frameworks to ensure consistency throughout the migration process.
• Conduct code reviews, performance testing, and production readiness validation.
• Create and maintain technical documentation, including migration strategies, dependencies, and operational guides.
• Work closely with business and application stakeholders to ensure seamless handoff and domain alignment.

• Data Processing:
  Apache Spark, PySpark, Hive, NiFi, Kafka, Trino, Iceberg
• Languages:
  
  Python, Scala, Java
• Data Storage & Formats:
  Parquet, ORC, Avro, S3, HDFS
• Orchestration:
  Airflow, DBT, Oozie
• Performance Tuning: YARN, Kubernetes, partitioning, caching
• Testing & Validation:
  Great Expectations, Deequ, SQL‑based validation
• Monitoring:
  Datadog, Grafana, Prometheus

• 4–8 years of experience in big data engineering, ETL modernization, or distributed system development.
• Proven experience migrating workloads from Hadoop‑based ecosystems (Cloudera, MAPR, Hortonworks) to open‑source or cloud‑native platforms.
• Deep understanding of data lake and distributed data architecture principles.
• Hands‑on experience with hybrid or cloud environments (AWS, Azure, or GCP).
• Familiarity with large enterprise or regulated environments preferred.

Brilliant Staffing, LLC is an Equal Opportunity Employer and encourages applications from all individuals regardless of race, color, religion, gender, gender identity, sexual orientation, national origin, disability, or veteran status.