Senior Data Engineer

Required

Skills and Experience:

• Strong programming skills in Python and Java.
• Hands-on experience with cloud platforms, preferably GCP.
• Experience working with Docker containers.
• Knowledge of environment management tools such as venv, pip, poetry, etc.
• Experience with orchestrators like Vertex AI pipelines, Airflow, etc.
• Data engineering and feature engineering techniques.
• Proficiency in Apache Spark, Apache Beam, or Apache Flink.
• Advanced SQL knowledge.
• Understanding of streaming concepts such as windowing, late arrival, triggers, etc.
• Hands-on experience with distributed computing.
• Experience in designing data architecture.
• Knowledge of storage and compute options and their appropriate use cases.
• Understanding of cluster and pipeline optimization strategies.
• Exposure to GCP tools for developing end-to-end data pipelines, including data ingestion from traditional databases and API sources.
• Business mindset to interpret data for BI and analytics.
• Experience with CI/CD pipelines, deployment methodologies, and Infrastructure as Code (e.g., Terraform).
• Good to have:
  Hands-on experience with Kubernetes.
• Good to have:
  Experience with vector databases like Qdrant.

Experience with GCP tools:

• Ingestion: Pub/Sub, Cloud Functions, App Engine, Kubernetes Engine, Kafka, Microservices.
• Processing: Cloud Dataproc, Cloud Dataflow, Apache Spark, Apache Flink.
• CI/CD: Bitbucket, Jenkins, Git Lab.