Data Engineer

Key Responsibilities
Technical Leadership & Ownership
Own the end-to-end data engineering architecture for large-scale AWS data platforms
Define and enforce data engineering standards, best practices, and governance frameworks
Lead design reviews, code reviews, and technical decision-making across teams
Act as the primary technical escalation point for complex data pipeline issues

ETL/ELT Design & Development
Design, build, and optimize scalable ETL/ELT pipelines using:
AWS Glue (Jobs, Workflows, Crawlers)
PySpark / Spark SQL, Snowflake, SnowsQL
Python- based data processing frameworks
Implement incremental processing, CDC, and data partitioning strategies
Develop reusable and modular data pipeline frameworks for enterprise use

Data Lake & Storage Management
Design and manage data lake architecture on  AWS (S3 + Apache Iceberg )
Implement ACID-compliant data layers using Iceberg
O ptimize storage formats (Parquet, ORC)  and data layouts for performance
Define and enforce data lifecycle, retention, and archival policies

Performance Optimization & Cost Efficiency
Tune Spark/Glue jobs for performance optimization (memory, partitioning, caching)
Optimize workloads for cost efficiency in AWS (compute, storage, I/O)
Monitor and improve pipeline SLAs, throughput, and latency metric

Data Governance & Quality
Implement data quality frameworks, validations, and reconciliation checks
Ensure compliance with data governance, lineage, and security standards
Work with cataloging tools (AWS Glue Data Catalog, etc.) for metadata management

Integration & Orchestration
Design and manage end-to-end orchestration workflows (Glue Workflows, Step Functions, Airflow if applicable)
Integrate data across multiple sources (RDBMS, APIs, streaming platforms, files)
Enable reliable, fault-tolerant, and restartable pipeline execution

Stakeholder Collaboration
Partner with business, analytics, and AI teams to understand data requirements
Collaborate with architects and Dev Ops teams for environment setup and automation
Provide technical guidance to junior engineers and team members

Team Leadership & Mentoring
Lead and mentor a team of data engineers
Drive skill development in Spark, AWS, and modern data architectures
Ensure adherence to Agile practices and timely delivery of milestones

Required Skills & Experience
Core Technical Skills
Strong experience in AWS Data Engineering stack:
AWS Glue, S3, Lambda, IAM, Cloud Watch
Advanced proficiency in:
PySpark / Apache Spark
Spark SQL
Python
Hands-on experience with Apache Iceberg / modern table formats
Deep understanding of ETL/ELT design patterns and data pipelines

Data Engineering Expertise

Experience with data lake and lakehouse architectures
Strong knowledge of data modeling (star/snowflake schemas)

Experience with batch and near real-time processing
Familiarity with file formats (Parquet, ORC, Avro)

Performance & Optimization
Proven experience in large-scale data processing (TB/PB scale)
Strong expertise in query optimization, partitioning, and indexing strategies

Dev Ops & Automation

Experience with CI/CD pipelines for data workflows
Knowledge of infrastructure as code (Cloud Formation/Terraform) is a plus
Familiarity with version control (Git) and deployment strategies

Preferred Skills (Good to Have)

Experience with data orchestration tools (Airflow, Step Functions)
Exposure to streaming frameworks (Kafka, Kinesis)
Knowledge of data security (encryption, masking, access control)
Experience supporting AI/ML data pipelines
Exposure to BI tools (Power BI, Tableau, Sigma)

Qualifications
Bachelor’s/Master’s degree in Computer Science, Engineering, or related field
8–12+ years of experience in data engineering, with 3+ years in a technical leadership role