Back Button

Data Engineer

We are seeking a skilled Data Engineer to design, build, and maintain scalable data pipelines and platforms that enable analytics, reporting, and real-time applications. The ideal candidate will have hands-on experience with both batch and stream processing, strong expertise across cloud data ecosystems (GCP, AWS, Azure), and proficiency in big data frameworks and modern data engineering tools. You will collaborate closely with data scientists, analysts, and product teams to ensure high-quality, reliable, and performant data solutions.

Key Responsibilities

• Design, develop, and maintain robust ETL/ELT data pipelines for batch and real-time streaming workloads using tools such as Apache Spark, Flink, Beam, Kafka, Kinesis, and AWS Glue.
• Implement and optimize scalable data processing on cloud platforms (GCP, AWS, Azure) leveraging services like Big Query, Snowflake, Databricks, Amazon Redshift, Delta Lake, and Google Cloud Storage/Amazon S3.
• Develop and maintain data lake and warehouse solutions, including schema design, dimensional modeling, and performance tuning for queries and data ingestion.
• Work with a variety of data formats (Parquet, ORC, Avro, JSON, CSV) and storage systems (HDFS, S3) to ensure efficient, compressed, and query-friendly datasets.
• Build and operate data streaming architectures and real-time pipelines, ensuring low-latency processing and exactly-once or at-least-once semantics as required.
• Author, schedule, and monitor workflows and DAGs using Apache Airflow or equivalent orchestration tools; implement alerting and observability for pipeline health.
• Implement data quality checks, monitoring, data lineage tracking, metadata management, and data catalog integration to support governance and compliance requirements.
• Optimize database performance and implement indexing, partitioning, and query optimization strategies for PostgreSQL, MySQL, Oracle, Cassandra, MongoDB, and No
  
  SQL stores.
• Containerize data applications using Docker and deploy on Kubernetes or managed services; use infrastructure as code with Terraform or Cloud Formation to provision reproducible environments.
• Collaborate with data scientists and analysts to provide reliable datasets and REST/JDBC data access for analytics, machine learning, and BI tools (Tableau, Looker, Power BI).
• Implement CI/CD pipelines, version control (Git), and automated testing for data infrastructure and ETL code to enable safe, repeatable deployments.
• Drive best practices in data modeling, security, access controls, and metadata hygiene; evangelize data governance, documentation, and cataloging across teams.
• Mentor junior engineers, participate in design reviews, and contribute to architectural decisions for scaling data platforms and pipelines.

Qualifications

• Bachelor's or Master's degree in Computer Science, Engineering, or related technical field, or equivalent practical experience.
• Proven experience as a Data Engineer or similar role building production data pipelines and platforms at scale.
• Proficiency in SQL and experience with one or more programming languages:
  Scala, Java, or Python (pandas, Num Py, Jupyter).
• Hands-on experience with big data frameworks and processing engines such as Apache Spark, Hadoop, Hive, Flink, and Apache Beam.
• Experience building and operating stream processing and messaging systems:
  Kafka, Kinesis, or equivalent.
• Solid experience with orchestration tools such as Apache Airflow and cloud-native ETL tools like AWS Glue; familiarity with Lambda and serverless patterns.
• Strong knowledge of cloud data services across GCP, AWS, and Azure, including Big Query, Snowflake, Databricks, Redshift, and managed storage (S3, GCS).
• Experience with data formats (Parquet, ORC, Avro, JSON, CSV) and storage systems (HDFS, S3) and optimized data layout strategies (Delta Lake, partitioning, compaction).
• Familiarity with relational and No
  
  SQL databases (PostgreSQL, MySQL, Oracle, MongoDB, Cassandra, Redis) and data warehousing concepts and dimensional modeling.
• Experience with CI/CD, Git, containerization (Docker), orchestration (Kubernetes), and infrastructure-as-code (Terraform, Cloud Formation).
• Skills in query optimization, performance tuning, monitoring (Prometheus, Grafana), and operationalizing data quality and observability.
• Knowledge of data governance, metadata management, data catalogs, lineage tracking, and privacy/compliance considerations.
• Experience integrating data for BI and analytics tools via JDBC/REST and familiarity with Tableau, Looker, or Power BI.
• Excellent problem-solving skills, strong communication and collaboration abilities, and a commitment to mentoring and knowledge sharing.