Senior Data Engineer

Hi,

We are currently seeking Senior AWS Data Engineer position @ TAO Digital Solutions.

We are seeking a Senior AWS Data Engineer to implement, operate, and continuously optimize a cloud-native OLAP and Lakehouse platform on AWS. While core architecture patterns are already defined, this role is responsible for correct implementation, operational excellence, observability, and cost-efficient operation of the data platform. The platform supports customer-facing analytics, self-service reporting, and future AI/ML workloads using Amazon Redshift, Apache Iceberg on S3, PySpark, and AWS-native services.

Key Responsibilities

• Implement and operate AWS-native OLAP and Lakehouse architecture (Redshift, S3, Iceberg)

• Build and operate Aurora MySQL data migration, replication, and CDC pipelines

• Develop and maintain PySpark-based ETL pipelines

• Implement Redshift ETL, views, and materialized views

• Actively optimize query performance and cost in Redshift and Iceberg

• Manage Iceberg tables, MERGE logic, partitioning, and schema evolution

• Orchestrate data pipelines using Airflow and/or AWS Step Functions

• Implement observability, data quality checks, alerts, and operational dashboards

• Enforce PHI masking, tenant isolation, and query guardrails

• Continuously optimize storage, compute usage, and query cost efficiency

Required Qualifications

• 10+ years of hands-on experience in Data Engineering

• Strong experience with Amazon Redshift, S3, and AWS Glue Data Catalog

• Mandatory experience with Apache Iceberg (MERGE, partitioning, schema evolution)

• Mandatory experience with PySpark for large-scale data transformations

• Experience with Aurora MySQL migration, replication, and CDC pipelines

• Hands-on experience with Airflow and/or AWS Step Functions

• Experience developing AWS Lambda-based data workflows

• Infrastructure-as-code experience using Terraform

• Advanced SQL and strong Python data engineering skills

• Strong experience optimizing analytics cost and query efficiency

What Success Looks Like

• Analytics workloads are reliable, observable, and cost-efficient

• Query costs in Redshift and Iceberg are predictable and well-controlled

• Data freshness, security, and tenant isolation are enforced by design

• The platform is stable, scalable, and ready for AI-native workloads

Interested candidates please forward your updated resume to the following email  (mohan.kali)