Java Tech lead ( Nifi, Airflow

Purple Drive

Java Tech Lead (Nifi, Airflow)

Share Phoenix, AZ, Raleigh, NC PDT

ROLE:
Java Tech Lead (Nifi, Airflow)

Description:

• Strong NiFi Expertise:
Deep understanding of Apache NiFi architecture, processors, controllers, and data flow management capabilities.

• Programming

Languages:

Proficiency in Java, Spring boot & API Microservices.

• Database and SQL:
Familiarity with databases, SQL, and data manipulation techniques.

• Big Data Technologies:
Kafka (Event driven & streaming architecture)

• Apachae Airflow – Workflow orchestration

• Design and Implementation:
Creating and implementing data flow solutions using NiFi, including defining data sources, destinations, and transformation processes.

• Ingestion and Transformation:
Managing data ingestion from various sources (e.g., files, databases, APIs) and transforming data into desired formats for downstream consumption.

• Routing and Enrichment:
Configuring NiFi to route data to appropriate destinations and enriching data with additional information (e.g., timestamps, location).

• Scalability and Performance:
Ensuring data flows are scalable and performant to handle varying data volumes and velocity.

• NiFi Cluster Management:
Maintaining and improving NiFi clusters, including configuration, optimization, and troubleshooting.

• Microservices Integration:
Developing and integrating microservices to support data conditioning, format validation, and transformation processes.

• Troubleshooting and Issue Resolution:
Identifying and resolving issues related to data flows, NiFi configurations, and data integration processes.

• Airflow Expertise:
Possessing deep knowledge of Airflow's architecture, including schedulers, executors (Celery, Kubernetes), and plugin development.

• Workflow Design and Development:
Designing and developing complex, modular, and reusable DAGs (Directed Acyclic Graphs) to automate data pipelines.

• Performance Optimization:
Identifying and addressing performance bottlenecks in Airflow environments and implementing best practices for orchestration and scheduling