Python Engineer, PySpark​/Spark Tuning, Azure DevOps​/Jenkins, Hive​/Impala; NJ

Overview

Start: ASAP
• Glider:
Candidate Identification Only
• 3 days Onsite Required
• Jersey City, NJ - NJ.

MUST INCLUDE CURRENT LOCATION AND INTENDED ON LOCATION RESUME.

• The Python Engineer designs and implements data ingestion and transformation jobs using Python and PySpark.
• They build reusable frameworks for data quality checks and integrate pipelines with CI/CD processes.
• The role requires performance tuning, secure coding practices, and strong debugging skills.
• Close collaboration with architects and Data Ops teams ensures robust and compliant solutions.

• The Python Engineer designs and implements data ingestion and transformation jobs using Python and PySpark.
• They build reusable frameworks for data quality checks and integrate pipelines with CI/CD processes.
• The role requires performance tuning, secure coding practices, and strong debugging skills. Close collaboration with architects and Data Ops teams ensures robust and compliant solutions.

• Strong Python (packaging, virtual environments), PySpark/Spark performance tuning.
• Experience with data ingestion (batch/stream), schema management, and error handling/retry logic.
• Test discipline: unit/integration tests, data quality assertions, reproducible pipelines.
• CI/CD (Azure Dev Ops/Jenkins), Git workflows, artifact versioning, release readiness.
• Experience on Cloudera/Hadoop (HDFS, Spark, Hive/Impala) and Databricks (clusters, jobs, notebooks, Delta).
• Observability: structured logging, metrics, tracing; debuggability in distributed contexts.
• Secure coding and handling of secrets, PII protection, and compliance considerations.
• Strong communication and collaborative work style; documentation of frameworks and usage patterns.
• Performance optimization: partitioning, caching, broadcast joins, memory tuning.
• Comfort with Agile ceremonies and iterative delivery.
• Min 5 yrs of experience