Data Engineer

Job Description:

Data Engineer Function:
Data & Analytics | GCP About the Role

We are looking for a skilled and motivated Data Engineer to join our data platform team. In this role, you will design, build, and maintain scalable data pipelines and infrastructure on Google Cloud Platform (GCP). You will work closely with data analysts, data scientists, and platform teams to ensure reliable, efficient, and cost-effective data movement and transformation across the organization

• Design, develop, and maintain robust batch and streaming data pipelines using Apache Airflow (Astronomer / Cloud Composer) and Cloud Scheduler
  .
• Build and manage real-time data ingestion pipelines using Confluent Kafka and Google Pub/Sub
  .
• Develop and optimize data processing jobs using Dataflow (Apache Beam) and Dataproc (PySpark).

• Design and manage data models, tables, and datasets in Big Query for performance and cost efficiency.
• Manage data storage in Google Cloud Storage (GCS) including partitioning, lifecycle policies, and access controls.

• Provision and manage cloud infrastructure using Terraform (Infrastructure as Code) for platform and ML model deployments.
• Build and maintain CI/CD pipelines for automated testing, deployment, and versioning of data pipelines - including schema and contract validation gates to ensure data integrity across environments.
• Support MLOps pipeline build-out
  , enabling reliable model training, versioning, and deployment workflows.
• Implement and manage storage tiering automation to optimize data retention, access patterns, and cost across GCS and Big Query.
• Deploy and manage containerized data services using Cloud Run
  .
• Manage source code and collaboration via Cloud Repository / Git Hub
  .
• Handle credentials and sensitive configurations securely using Secret Manager
  .

• Design and implement event-driven data architectures using Confluent Kafka and Google Pub/Sub
  .
• Ensure low-latency, high-throughput data delivery across systems.

• Partner with Data Analysts and Data Scientists to understand data needs and deliver reliable datasets.
• Implement data quality checks, monitoring, and alerting across pipelines.
• Document pipeline architecture, data flows, and operational runbooks.

Astronomer Apache Airflow, Cloud Composer, Cloud Scheduler

Astronomer Apache Airflow, Cloud Composer, Cloud Scheduler

Dataproc, Py Spark

Dataflow (Apache Beam), Confluent Kafka, Pub/Sub

Big Query

Google Cloud Storage (GCS)

Cloud Run

Terraform

CI/CD Pipelines (Cloud Build / Git Hub Actions / Jenkins)

Secret Manager

Cloud Repository, Git

Python, PySpark, SQL

ML pipeline orchestration, model deployment, storage tiering

• Strong hands-on experience with GCP data services (Big Query, Dataflow, Dataproc, Pub/Sub, GCS).
• Proficiency in Python and SQL for data pipeline development and transformation.
• Experience with Py Spark for large-scale distributed data processing.
• Hands-on experience with Apache Airflow (Astronomer or Cloud Composer).
• Working knowledge of Terraform for infrastructure provisioning and ML model deployments.
• Experience building and maintaining CI/CD pipelines including schema and contract validation gates.
• Familiarity with MLOps practices and supporting model deployment pipelines.

Familiarity with Kafka or event-driven streaming architectures.