PySpark Developer

Location: Irving, TX
Salary: $ USD Annually - $ USD Annually
Description:
Job Description - PySpark Developer

Location: Irving, TX
Experience: 5-10 years

Overview

We are seeking an experienced PySpark Developer with a strong background in data engineering to design, build, and optimize scalable data pipelines. The ideal candidate will have deep expertise in Apache Spark
, hands-on PySpark development
, and experience working with large, complex datasets. You will collaborate closely with business analysts and stakeholders to deliver high-quality data solutions that support analytics and business intelligence.

Responsibilities

• Develop, optimize, and maintain data pipelines and ETL workflows using Apache Spark and PySpark.
• Apply advanced knowledge of Spark core concepts
  , performance tuning, and industry best practices.
• Translate business requirements into scalable and reliable data transformations.
• Work with analysts and stakeholders to understand data requirements and deliver innovative solutions.
• Write and maintain unit tests
  , follow OOP principles, and ensure high code quality.
• Develop complex SQL queries to extract insights from large-scale datasets.
• Schedule, orchestrate, and monitor data transformation jobs based on business needs.
• Perform root-cause analysis on pipeline failures, resolve data quality issues, and implement corrective fixes.
• Ensure data integrity, reliability, and availability across the data ecosystem.

Required Skills & Experience

• 5-10 years of hands-on experience in Py Spark and Apache Spark development.
• Strong understanding of Spark RDDs, Data Frames, Spark SQL, and performance optimization.
• Expertise in Python with ability to implement complex data transformations.
• Solid SQL skills with experience handling large datasets.
• Knowledge of object-oriented programming, unit testing, and debugging.
• Experience with scheduling tools (Airflow, Cron, or similar).
• Strong analytical and troubleshooting skills for data pipeline issues.

By providing your phone number, you consent to: (1) receive automated text messages and calls from the Judge Group, Inc. and its affiliates (collectively "Judge") to such phone number regarding job opportunities, your job application, and for other related purposes. Message & data rates apply and message frequency may vary. Consistent with Judge's Privacy Policy, information obtained from your consent will not be shared with third parties for marketing/promotional purposes.

Reply STOP to opt out of receiving telephone calls and text messages from Judge and HELP for help.

Contact:

This job and many more are available through The Judge Group. Please apply with us today!