AWS Data Engineer

Experience:

5+ years

Work mode:
Hybrid preferred or Remote

Freelances who can work for 5 to 6 hours in a day

We are looking for a hands‑on engineer with strong expertise in AWS‑based data pipelines to help enhance alerting systems, optimize compaction jobs, strengthen data sanity frameworks, and drive cost optimization initiatives. The ideal candidate should have experience working with large‑scale data systems, observability frameworks, and AWS‑native services.

• Alerts & Monitoring Refinement
• Design and implement batch alerts for data processing pipelines.
• Refine existing alerting mechanisms to improve signal‑to‑noise ratio and operational efficiency.
• Integrate new job dashboards and alerting systems with existing monitoring frameworks.
• Migrate monitoring workloads away from Amazon Cloud Watch to more cost‑efficient alternatives while maintaining observability standards.
• Compaction Job Optimization
• Review and re‑evaluate compaction job changes previously implemented.
• Implement improvements to reduce storage and compute costs.
• Optimize data lifecycle, partitioning, and storage formats (e.g., Parquet, ORC).
• Improve job scheduling and execution efficiency in AWS data pipelines.
• Data Sanity Framework Enhancement
• Revisit the existing data sanity/validation framework.
• Identify gaps in data validation, quality checks, and anomaly detection.
• Implement automated data quality checks across ingestion and transformation layers.
• Establish measurable SLAs and reporting for data reliability.
• Cost Optimization
• Identify cost drivers across AWS data infrastructure.
• Optimize compute (e.g., Amazon EC2), storage (e.g., Amazon S3), and analytics services (e.g., Amazon EMR).
• Improve resource utilization, implement right‑sizing strategies, and introduce lifecycle policies.
• Provide periodic cost analysis and recommendations with measurable impact.

• 5+ years of experience in AWS‑based data engineering.
• Strong hands‑on experience with AWS data services such as:
  • Amazon S3
  • Amazon EMR
  • AWS Glue
  • Amazon EC2
  • Amazon Cloud Watch
• Experience building and maintaining batch and streaming data pipelines.
• Strong understanding of data compaction strategies and storage optimization.
• Experience in observability, monitoring, and alert tuning.
• Proven track record in AWS cost optimization initiatives.
• Strong SQL and scripting skills (Python/Scala preferred).

• Experience with large‑scale time‑series or telemetry data systems.
• Exposure to data validation frameworks and automated quality checks.
• Familiarity with Click House or columnar data stores.
• Experience working in high‑scale, production‑grade environments.