Senior Python Big Data Engineer — AWS, Spark, EMR
Listed on 2026-10-29
-
Software Development
Data Engineering, Python, AWS
We are seeking an experienced Python Big Data Engineer with expertise in designing, developing, and optimizing large-scale data processing solutions. The ideal candidate will have extensive experience with Python, AWS cloud services, big data technologies, and data engineering best practices to build scalable, high-performance data pipelines and analytics solutions.
Key Responsibilities
• Design, develop, and maintain scalable big data applications using Python.
• Build and optimize data processing pipelines using Hadoop, EMR, Spark, and Hive.
• Develop and support cloud-native data solutions using AWS services.
• Design and implement data ingestion, transformation, and processing workflows.
• Develop and maintain integrations using APIs and SQL queries.
• Perform large-scale data manipulation, transformation, analysis, and visualization.
• Analyze large datasets to extract meaningful business insights.
• Design, implement, and support CI/CD pipelines for automated deployment.
• Collaborate with cross-functional teams to deliver scalable and reliable data solutions.
• Apply analytical and problem-solving skills to identify and resolve complex technical challenges.
Required Qualifications
• 10+ years of hands-on Python development experience for big data applications.
• Extensive experience implementing scalable data processing pipelines using Hadoop, Amazon EMR, Spark, and Hive.
• 5+ years of development experience with AWS services including EMR, Lambda, Event Bridge, Step Functions, Redshift, Amazon S3, and AWS Glue.
• 5+ years of experience developing solutions involving API integrations, SQL queries, Spark, data manipulation, data transformation, analysis, and visualization.
• Strong experience analyzing large datasets and extracting meaningful insights.
• Hands-on experience implementing CI/CD pipelines using Git Lab, Terraform, or similar tools.
• Strong analytical, critical thinking, and problem-solving skills.
• Ability to collaborate effectively across cross-functional and matrixed teams.
Preferred Qualifications
• Experience with infrastructure as code (IaC).
• Experience optimizing large-scale cloud-based data engineering solutions.
(If this job is in fact in your jurisdiction, then you may be using a Proxy or VPN to access this site, and to progress further, you should change your connectivity to another mobile device or PC).