Senior Data Engineer

Job Title:

Senior Data Engineer Lakehouse Platform

Experience: 8+ years
Employment Type: Contractual
Work Mode: 100% Remote
Working Hours: 6:00 PM – 3:00 AM IST (US Shift)

We are looking for an experienced Senior Data Engineer to design, build, and scale a production-grade data lakehouse platform on AWS. The role involves building a reliable, governed, and scalable data foundation to support enterprise analytics, financial reporting, and AI-driven applications.

The ideal candidate will have strong hands‑on expertise in AWS data services, Apache Iceberg, dbt, SQL Server CDC pipelines, and modern data architecture, with a focus on data accuracy, reliability, performance, and cost optimization.

• Design and scale a Medallion Lakehouse Architecture (Bronze, Silver, and Gold) using Apache Iceberg, Amazon S3, Athena, Redshift Serverless, dbt Cloud, and AWS Lake Formation.
• Build and maintain AWS DMS-based Change Data Capture (CDC) pipelines from SQL Server databases.
• Develop reliable SQL transformations for complex business processes, including billing, invoicing, time tracking, and financial reporting.
• Implement SCD Type 2, snapshots, incremental processing, and historical data tracking.
• Design scalable multi-tenant data architectures with appropriate tenant isolation, partitioning, and skew management.
• Manage dbt Cloud workflows, including semantic layers, dbt Mesh, Metric Flow, testing, and incremental models.
• Build governed data models and metrics to support analytics and AI-agent consumption, including MCP integrations where applicable.
• Establish data quality checks, reconciliation processes, freshness SLAs, observability, and data lineage.
• Optimize query performance, storage costs, Iceberg compaction, and data processing efficiency.
• Manage Iceberg operations, including snapshot expiry, time travel, and partition evolution.
• Collaborate with application, analytics, and AI teams to translate business requirements into reliable data products.
• Troubleshoot production issues and ensure data accuracy, availability, and consistency.

• 10+ years of experience building and managing production‑grade data platforms.
• At least 3 years of experience with lakehouse architectures using Apache Iceberg, Delta Lake, or Apache Hudi.
• Strong hands‑on experience with AWS DMS and CDC pipelines from SQL Server databases.
• Advanced dbt Cloud skills, including semantic layer, dbt Mesh, Metric Flow, snapshots, SCD2, testing, and incremental models.
• Production experience with Amazon S3, Athena, Redshift Serverless, and AWS Lake Formation.
• Strong SQL Server and T‑SQL skills, including the ability to understand complex legacy application logic.
• Expertise in dimensional modeling, data warehousing, and Kimball/Inmon modeling approaches.
• Experience with multi‑tenant SaaS data platforms at scale.
• Hands‑on knowledge of Iceberg compaction, snapshot management, partition evolution, and time travel.
• Proven experience with financial data reconciliation and business‑critical data accuracy.
• Strong analytical, troubleshooting, and performance‑tuning skills.

• Experience exposing governed metrics or dbt semantic layers to AI agents through MCP.
• Familiarity with AWS Glue, Step Functions, Event Bridge, MWAA, and EMR Serverless.
• Experience with zero‑ETL integrations, such as Aurora to Redshift or Open Search.
• Experience in fintech, legal technology, or other regulated industries.
• Knowledge of query‑plan analysis, Redshift sort keys, and Iceberg file‑size optimization.
• Exposure to LLM‑powered applications and AI‑enabled data platforms.

• Contractual engagement with a fully remote working arrangement.
• Candidates must be available to work from 6:00 PM to 3:00 AM IST
  .
• Candidates must be willing to commit to a minimum engagement of one year
  .
• Immediate joiners or candidates with a short notice period will be preferred.
• Candidates must be comfortable working on a contractual basis and independently owning production data infrastructure.