GCP Data Engineer

Mandatory: Big Query, Big Table, Google Cloud Storage, Pub Sub, Data Fusion, Dataflow, Dataproc

• Google Cloud Platform (GCP) Tools (Preferred):
  Big Query, Cloud Storage, Dataflow, Cloud Functions, Pub/Sub, Cloud Run, Cloud Composer (Airflow), Cloud Spanner, Bigtable.
• Container Orchestration:
  Kubernetes (preferred on GKE) and Helm for managing and deploying containerized applications.
• CI/CD and Automation:
  Jenkins for building CI/CD pipelines to automate deployment and testing of data pipelines.
• Programming
  
  Languages:
  
  Proficient in Python for data processing and automation, SQL for querying and data manipulation. Experience with Java is a plus.
• Dev Ops Tools:
  Familiarity with Terraform or Deployment Manager for Infrastructure as Code (IaC) to manage GCP resources.
• Monitoring and Logging:
  Experience with Cloud Monitoring, Datadog, or other monitoring solutions to track pipeline performance and ensure operational efficiency.
• Data Engineering:
  Expertise in ETL/ELT pipelines, data modeling, and data integration across large datasets.
• Strong understanding of data warehousing and real-time data processing workflows.
• Strong communication skills to work effectively with cross-functional teams and mentor junior developers. Proven ability to lead in an Agile environment.
• 3+ years of experience as a data engineer, with hands-on experience in Kubernetes, Helm, Python and Jenkins.
• Strong experience building and optimizing data pipelines and services in any cloud platform.
• Proficiency in Python and SQL. Familiarity with Java and Docker is a plus.

• The GCP Data Engineer will work iteratively on the cloud platform to design, develop and implement scalable, high performance solutions that offer measurable business value to customers.
• Data Solutions:
  Design and implementation of data pipelines using GCP services.
• Manage Deployments with Helm and Kubernetes:
  Oversee containerized deployments of data applications on Kubernetes Engine using Helm for package management, ensuring efficient orchestration of services.
• Automation with Jenkins:
  Implement CI/CD pipelines using Jenkins to automate data pipeline deployment, testing, and integration with other services, ensuring quick iterations and deployments.
• Develop and Optimize Pipelines:
  Write efficient Python and SQL scripts to build data pipelines and ETL/ELT processes. Continuously monitor and optimize data workflows for performance and cost-effectiveness.
• Data Integration and Orchestration:
  Design workflows to integrate data from various sources using GCP services, and orchestrate complex tasks with Cloud Composer (Apache Airflow).
• Security and Compliance:
  Ensure data security and compliance by implementing IAM policies, encryption, and other security measures in GCP, adhering to best practices for handling sensitive data.
• Collaborate Across Teams:
  Work closely with application developers, data architects, and business stakeholders to define and deliver robust data-driven solutions. Provide technical leadership and ensure alignment between business and technical objectives.
• Monitoring and Logging:
  Set up monitoring, logging, and alerting using Cloud Monitoring (formerly Stackdriver), Datadog, or other tools to ensure visibility into pipeline performance and quickly identify and resolve issues.