Lead Data Engineer - GCP

Responsibilities

• Design and architect scalable, secure, and high-performance data platforms on GCP
• Lead the development of large-scale batch and streaming data pipelines using PySpark and Python
• Build and optimize orchestration frameworks using Apache Airflow/Cloud Composer
• Design enterprise-grade data models and transformation frameworks using dbt
• Implement scalable data warehouse and lakehouse solutions using Big Query
• Define architecture standards, engineering best practices, and governance frameworks
• Drive performance optimization, scalability, and cloud cost optimization initiatives
• Collaborate with Architects, Product Owners, Analysts, and business stakeholders to translate business requirements into technical solutions
• Lead technical design discussions, code reviews, and architecture reviews
• Implement observability, monitoring, lineage, and data quality frameworks
• Mentor junior engineers and provide technical leadership to the team
• Ensure compliance with security, governance, and regulatory standards
• Support CI/CD and Infrastructure as Code implementations

• 6–9 years of experience in Data Engineering
• Strong expertise in GCP data ecosystem and cloud-native architectures
• Deep hands-on experience with:
  • Python
  • Py Spark
  • SQL
  • Apache Airflow / Cloud Composer
  • dbt
  • Big Query
• Strong understanding of data architecture patterns including:
  • Lakehouse Architecture
  • Medallion Architecture
  • Data Mesh concepts
  • Batch and Streaming architectures
• Experience with GCP services such as:
  • Big Query
  • Dataproc
  • Cloud Storage
  • Pub/Sub
  • Dataflow
  • Composer
  • IAM
• Expertise in data modeling, partitioning, clustering, and query optimization
• Strong understanding of distributed systems and scalable data processing
• Experience with CI/CD, Git Ops, Terraform, and containerization technologies
• Excellent stakeholder management and technical leadership skills