Data Engineer

Data Engineer

Job Description

• Orchestration:
  Apache Airflow, Apache Ni Fi
• Programming:
  Java (Core), Python (for Airflow), Unix Shell Scripting
• Big Data/Storage:
  Apache Spark, MinIO, AWS S3
• Security: SSL/TLS, Certificate Management, IAM, Java Keystores
• OS:
  Linux/Unix (RHEL/Ubuntu)

• Pipeline Orchestration:
  Design and develop complex, reusable DAGs in Apache Airflow to automate data workflows, scheduling, and monitoring across the enterprise.
• Data Ingestion & Flow:
  Create and optimize high-volume data streams using Apache NiFi, leveraging custom processors and controller services for diverse data sources and sinks.
• Custom Development:
  Utilize Java to develop custom NiFi processors or troubleshoot the core NiFi framework, ensuring the platform meets specific architectural requirements.
• Object Storage Management:
  Implement and manage data storage solutions using MinIO and AWS S3, ensuring high availability and efficient data retrieval patterns.
• Large-Scale Processing:
  Develop and maintain Apache Spark jobs for heavy-duty data transformations, integrating them into NiFi and Airflow orchestration layers.
• Security & Certificate Management:
  Secure NiFi clusters and data flows using TLS/mTLS, managing Java Key Stores (JKS), Trust Stores, and SSL certificate life cycles to ensure data-in-motion security.
• System Administration:
  Perform environment setup, performance tuning, and troubleshooting within Unix/Linux environments, including shell scripting for task automation.
• Cloud Integration:
  Deploy and manage data infrastructure components on AWS, utilizing IAM for access control and integrating cloud-native services with hybrid data pipelines.
• Monitoring & Optimization:
  Establish robust logging and alerting for data pipelines to proactively identify bottlenecks, ensuring 99.9% reliability of data delivery.

Salary Range - $100,000-$140,000 a year