Big Data Developer With GCP

• We are seeking an experienced Big Data Developer to join our client Amex. The ideal candidate will have strong expertise in Big Data technologies, Spark, Python, and GCP (Google Cloud Platform) to design, develop, and optimize large-scale data processing pipelines and enterprise applications.
• Key Responsibilities
• Design and develop scalable data pipelines and ETL processes using Spark and Python.
• Work on Big Data ecosystems for handling large datasets across distributed systems.
• Deploy, manage, and optimize applications on Google Cloud Platform (GCP) services like Big Query, Dataproc, Dataflow, and Pub/Sub.
• Collaborate with business stakeholders and data engineers to deliver high-quality data solutions.
• Ensure data quality, reliability, and security across applications.
• Participate in Agile/Scrum ceremonies and contribute to sprint deliverables.
• Troubleshoot and resolve performance issues in data pipelines and distributed systems.
• Required Skills
• Strong hands-on experience in Big Data technologies (Hadoop/Spark ecosystem).
• Proficiency in Apache Spark (PySpark, Data Frames, RDDs).
• Strong Python programming skills for data engineering and automation.
• Hands-on experience with GCP services (Big Query, Dataflow, Dataproc, Pub/Sub, Cloud Storage).
• Solid understanding of ETL frameworks, data ingestion, and data transformation.
• Knowledge of SQL and database optimization techniques.
• Experience with CI/CD pipelines, Git, Jenkins is a plus.
• Good problem-solving and communication skills.

Qualifications

Bachelor’s or Master’s degree in Computer Science, Engineering, or related field.

5+ years of professional experience in Big Data development and data engineering.

• Prior experience in financial domain projects is a plus