Lead Data Engineer

Responsibilities

• Design, build, and maintain efficient, reusable, and reliable architecture and code for data pipelines and data applications on AWS.
• Build robust data ingestion pipelines (from on-prem to AWS and within AWS) using AWS services such as Glue, Redshift, S3, Lambda, EMR/Spark, Kinesis, and SQS.
• Develop and manage ETL/ELT processes to collect, process, and store data from multiple sources, ensuring data quality, integrity, and security.
• Architect and implement end-to-end data solutions (ingestion, storage, integration, processing, access) on AWS, with a focus on data lakes and data warehouses.
• Participate in the architecture and system design discussions for high-scale data engineering projects.
• Independently perform hands‑on development, unit testing, and participate in code reviews to ensure adherence to best practices.
• Implement serverless applications using AWS Lambda, API Gateway, Step Functions, and other AWS technologies.
• Migrate data from traditional relational databases, file systems, and APIs to AWS-based data lakes (S3), RDS, Aurora, and Redshift.
• Implement high‑velocity streaming solutions using Amazon Kinesis, SQS, and Kafka (preferred).
• Architect and implement CI/CD strategies for enterprise data platforms.
• Collaborate with product, operations, QA, and cross‑functional teams throughout the software development cycle.
• Stay abreast of new technology developments, implement POCs for new tools/technologies, and onboard them for real‑world use cases.
• Identify and resolve performance issues and continuously optimize for cost, reliability, and scalability.

• Required Qualifications:
• - Bachelor’s degree in Computer Science, Software Engineering, MIS, or equivalent combination of education and experience.
• - 5+ years of experience implementing and supporting data lakes, data warehouses, and data applications on AWS for large enterprises.
• - Strong programming experience with Python, Shell scripting, and SQL.
• - Solid experience with AWS services:
  Cloud Formation, S3, Athena, Glue, EMR/Spark, RDS, Redshift, DynamoDB, Lambda, Step Functions, IAM, KMS, Secrets Manager.
• - Experience in serverless application development and data pipeline orchestration.
• - Experience in system analysis, design, development, and implementation of data ingestion pipelines in AWS.
• - Knowledge of ETL/ELT, data modeling, and big data technologies.
• - Familiarity with data warehousing concepts and cloud‑based architecture.
• - Strong problem‑solving skills and attention to detail.
• - Excellent communication and teamwork abilities.
• Preferred Qualifications:
• - Experience with additional AWS services: API Gateway, Elastic Search, SQS.
• - Experience with infrastructure‑as‑code tools (e.g., Terraform, Cloud Formation).
• - Experience with Dev Ops practices and CI/CD pipelines.
• - Experience implementing end‑to‑end streaming solutions (Amazon Kinesis, SQS, Kafka).
• - AWS Solutions Architect or AWS Developer Certification preferred.
• - Understanding of Lakehouse/data cloud architecture.
• - Knowledge of data governance and compliance standards.

Demonstrates expertise in designing and implementing data pipelines and applications on AWS, with a strong focus on data lakes, data warehouses, and ETL/ELT processes. Proficient in utilizing AWS services and programming languages to ensure data quality, integrity, and security while optimizing performance and scalability.

• AWS Data Pipeline Development
• ETL/ELT Process Management
• Python Programming
• Data Lake Architecture
• AWS Solutions Architect Certification

• Python
• SQL
• Shell Scripting
• ETL/ELT
• Data Modeling
• Big Data Technologies
• Data Ingestion Pipelines
• Data Warehousing
• Serverless Application Development
• Streaming Solutions

• Problem‑Solving
• Attention to Detail
• Communication
• Teamwork

• AWS Solutions Architect Certification
• AWS Developer Certification

• Data Governance
• Compliance Standards
• Cloud‑Based Architecture
• Infrastructure‑as‑Code
• Dev Ops Practices

• AWS Glue
• AWS Redshift
• AWS S3
• AWS Lambda
• AWS EMR/Spark
• AWS Kinesis
• AWS SQS
• Cloud Formation
• Terraform
• API Gateway