Lead Cloudera Streaming Architect

The Lead Cloudera Consultant (Solution Architect) will design, build, and optimize real-time streaming pipelines using the full Cloudera Data Platform (CDP) streaming stack.

This role requires deep hands-on expertise with NiFi, Kafka, Flink, Kudu/Impala, and SQL Stream Builder (SSB) to deliver mission-critical, high-volume Data-in-Motion solutions.

The consultant will architect and implement real-time ingestion, transformation, CDC patterns, and integration pipelines, while guiding engineering teams on best practices across the streaming ecosystem.

• Architect and build end-to-end real-time pipelines using NiFi, Kafka, Flink, Kudu/Impala, and SSB.
• Own architectural decisions, design patterns, schema evolution strategies, state management, and exactly-once delivery.
• Develop advanced NiFi flows including JDBC/DBCP services, record processors, stateful processors, schema registry integration, and high-volume ingestion patterns.
• Build and optimize Flink SQL and Data Stream API jobs including event-time windows, watermarks, checkpointing, and state backends.
• Design and tune Kudu tables including primary keys, partition strategies, distribution, upserts, deletes, and merges.
• Build, deploy, and monitor streaming SQL jobs using SQL Stream Builder (SSB).
• Deliver core real-time use cases including NiFi?
  
  Snowflake?
  
  Impala/Kudu ingestion, Kafka?
  
  Flink streaming, Flink?
  
  Kafka with exactly-once semantics, and CDC ingestion.
• Tune NiFi, Kafka, and Flink clusters to optimize throughput, reliability, and latency.
• Implement schema governance, back-pressure controls, error-handling frameworks, and replay strategies.
• Collaborate with platform engineering teams to optimize deployments and configurations.
• Provide architectural guidance, documentation, and mentorship to engineering teams.

• 8+ years of experience in data engineering or streaming system development.
• 3–5+ years of hands-on experience with Cloudera CDP/CDF streaming components.
• Strong production-grade experience with NiFi, including advanced processors, controller services, CDC patterns, and Snowflake/Kudu ingestion.
• Hands-on expertise with Kafka including topic design, consumer groups, schema registry, partition strategies, and high-volume streaming.
• Strong proficiency with Flink SQL and Data Stream API including event-time processing, windows, watermarks, checkpointing, and exactly-once semantics.
• Experience with Kudu table design, schema changes, upserts, deletes, merges, and Impala querying.
• Strong knowledge of SQL Stream Builder, job creation, connectors, and Flink SQL job deployment in CDP.
• Experience with CDC ingestion using NiFi, Flink CDC, or SSB including late events, schema evolution, deletes, and incremental key management.
• Strong SQL and distributed systems fundamentals.
• Experience in high-volume industries such as financial services, healthcare, or telecom is preferred.

• Experience running NiFi, Kafka, or Flink on Kubernetes.
• Snowflake ingestion experience including staging and COPY INTO patterns.
• Experience with Debezium for CDC.
• Experience with CI/CD for data pipelines.
• Security knowledge including Kerberos, Ranger, and Atlas.