Data Engineer

Key Skills & Technologies

• Programming
  
  Languages:
  
  Python (primary), SQL
• Cloud Platforms: AWS (S3, Glue, Lambda, Redshift, EC2, EMR)
• Data Tools:
  Apache Spark, Pandas, PySpark, Airflow
• Databases:
  Postgre
  
  SQL, MySQL, No
  
  SQL (e.g., Dynamo
  
  DB)
• ETL & Workflow Orchestration: AWS Glue, Apache Airflow
• Version Control:
  Git
• Dev Ops & CI/CD:
  Basic understanding of CI/CD pipelines and infrastructure as code (e.g., Terraform, Cloud Formation)

• Data Pipeline Development - Design, build, and maintain scalable and reliable data pipelines to ingest, process, and transform data from various sources.
• Data Integration & Management - Integrate structured and unstructured data from internal and external systems.
• Ensure data quality, consistency, and availability across platforms.
• Cloud-Based Data Engineering - Leverage AWS services (e.g., S3, Lambda, Glue, Redshift, EMR) to build cloud-native data solutions.
• Optimize cloud resources for performance and cost-efficiency.
• Programming & Automation - Use Python for data manipulation, ETL workflows, and automation of data tasks.
• Develop reusable scripts and modules for data processing.
• Collaboration & Stakeholder Engagement - Work closely with data scientists, analysts, and business teams to understand data needs.
• Translate business requirements into technical solutions.
• Monitoring & Optimization - Monitor data pipelines and troubleshoot issues proactively.
• Continuously improve performance, scalability, and reliability of data systems.