AWS Data Engineer

Technical Skills

• AWS
• Glue
• SNS/SQS
• Python
• Py‑Spark
• Data Lake
• Cloud Watch
• Cloud Trail
• DB Design
• SQL

• Lead the team technically to complete milestones on time.
• Understand complete requirement, create Architecture and update all stakeholders.
• Create POCs.
• Manage delivery/release to customer.
• Develop Services to enable data ingestion from and synchronization with system which exposes required data access mechanisms ensuring near‑real‑time updates.
• Ingest data from multiple sources using the python and other ETL tools.
• Design and implement an event‑driven architecture using AWS Event Bridge, Kafka, or SNS/SQS for real‑time data streaming.
• Design, implement, and maintain scalable data pipelines that integrate both on‑prem and AWS cloud environments.
• Develop efficient Python scripts and applications using libraries like pandas, Num Py, etc., to handle and process large datasets.
• Work with various No
  
  SQL databases (e.g., MongoDB, Cassandra, DynamoDB) to support high‑performance data storage and retrieval.
• Develop and deploy applications in a cloud‑native architecture, leveraging modern cloud technologies for scalability and resilience.
• Continuously monitor data workflows and systems, troubleshoot issues, and optimize performance for reliability and scalability, transitioning existing pipeline to MSSQL server.
• Collaborate with the business application owner on the existing data architecture, including data ingestion, data pipelines, business logic, data consumption patterns, and analytics requirements.
• Design and document the target data architecture, pipelines, processing and analytics architecture.
• Identify opportunities for optimization and consolidation.
• Collaboration with data team on decomposition of business logic and data transformation patterns.