GCP Data Engineer

We are looking for a talented Data Engineer with strong expertise in real-time data processing and scalable data platforms. In this role, you will design, build, and optimize robust data pipelines that power analytics and machine learning use cases. You’ll work with cutting-edge technologies such as Apache Spark, Kafka, and cloud-based solutions (primarily GCP) to deliver high-quality data at scale.

• Design & Development: Build, maintain, and enhance ETL/ELT pipelines for both real-time and batch ingestion using Spark (PySpark/Scala) and streaming frameworks.
• Streaming & Messaging: Implement and manage event-driven architectures using technologies such as Apache Kafka, Google Pub/Sub, or similar tools.
• Scalable Data Infrastructure: Architect and optimize scalable data platforms including data lakes, data warehouses (e.g., Big Query), and streaming systems.
• Performance Optimization: Continuously improve performance of Spark jobs, SQL workloads, and data processing workflows for speed, reliability, and cost efficiency.
• Data Quality & Reliability: Develop and enforce data quality frameworks, validation checks, and monitoring/alerting systems to ensure data integrity and trustworthiness.
• Collaboration & Best Practices: Work closely with data scientists, analysts, and platform engineers to align data delivery with business needs and industry best practices.
• Automation & Orchestration: Build automated workflows and scheduling using tools like Airflow or equivalent.

• Programming
  
  Languages:
  
  Strong proficiency in Python and SQL; experience with Scala or Java is a plus.
• Big Data Ecosystem: Hands-on experience with Apache Spark (including Spark SQL, Data Frames, and Streaming).
• Streaming Technologies: Demonstrated experience with event streaming platforms such as Apache Kafka, Google Pub/Sub, or equivalent.
• Cloud Platforms: Experience with Google Cloud Platform (GCP) services; familiarity with Azure data services is a bonus.
• Data Platforms: Understanding of data warehousing concepts and tools (e.g., Big Query, Snowflake, Redshift) and No
  
  SQL databases.
• Orchestration & Dev Ops: Exposure to workflow orchestration (Airflow), containerization (Docker), and orchestration (Kubernetes) adds strong value.
• Analytical Thinking: Strong problem-solving skills with the ability to work independently and in collaborative environments.