AWS Lakehouse Data Engineer

Work Model:
Remote – offsite

• Build and operate data pipelines (batch and streaming) from APIs, relational databases, file drops, event streams, and external partners.
• Design, implement, test, and optimize ETL/ELT pipelines using Python and PySpark to produce analytics-ready datasets for reporting, visualization, and machine learning.
• Implement incremental processing, change data capture (CDC), data contracts, schema validation, and reusable transformation frameworks.
• Improve pipeline reliability through automated testing, orchestration, monitoring, retries, and operational runbooks.
• Design and implement a Delta Lakehouse-style data platform on AWS using native services to provide Databricks-like capabilities for data engineering, analysis, and data visualization.
• Build and manage a scalable lakehouse on Amazon S3 using Apache Iceberg and open columnar formats such as Apache Parquet.
• Implement SQL-like table reliability features including ACID transactions, schema evolution, snapshot isolation, and time travel using Apache Iceberg.
• Enable fast, interactive queries of lakehouse data via AWS-native services like Amazon Athena, Amazon EMR, AWS Glue, and Amazon Redshift where appropriate.
• Optimize performance and cost through partitioning, file sizing, caching, lifecycle policies, and separating compute from storage.
• Establish standardized environments for development, testing, and production with consistent configuration and controlled promotion.
• Implement data governance, access control, lineage, and quality measures utilizing AWS-native services including AWS Lake Formation, AWS Glue Data Catalog, IAM, KMS.
• Create a metadata repository with cataloging, ownership, classification, tagging, and discoverability features.
• Enable end-to-end data lineage for audit and regulatory compliance.
• Apply policy-based access, least privilege, data classification, retention, encryption, and secure handling controls.
• Build data quality checks for freshness, completeness, validity, and anomaly detection, and publish SLA/SLO metrics.
• Automate AWS provisioning with Infrastructure as Code (IaC), develop CI/CD pipelines for data components, and ensure platform observability.
• Work collaboratively with cross-functional teams and maintain high-quality engineering documentation.

• Bachelor's degree in Engineering, Information Technology, Computer Science, Data Engineering, or related field, or four (4) years of equivalent practical experience.
• Six (6) years of relevant experience.
• Hands-on experience building AWS-native data lake or lakehouse architectures on Amazon S3.
• Strong experience developing production ETL/ELT pipelines with Python and PySpark, including data modeling, transformation, and performance tuning.
• Hands-on experience with Apache Iceberg, including ACID transactions, schema evolution, time travel, and query optimization.
• Advanced SQL skills supporting analytical workloads, reporting, and data visualization.
• Proven experience with data governance, cataloging, lineage, and access control using AWS services.
• Knowledge of AWS security fundamentals: IAM, KMS, secrets management, network security, logging, SDLC.
• Proven experience with Infrastructure as Code (IaC) and operating data platforms across environments.
• Experience with CI/CD pipelines for data workflows with testing, deployment, environment promotion, and rollback.
• Troubleshooting distributed data workloads, performance optimization, and cost management skills.
• Excellent collaboration and communication skills to coordinate with cross-team stakeholders.

• Experience with Databricks, Delta Lake, migrating workloads to AWS-native services, and Apache Iceberg.
• Familiarity with AWS Step Functions, MWAA,…