Sr. Software Engineer, Data Streaming Systems

Sr. Software Engineer, Data Streaming Systems Summary

The Applied Intelligence Data Engineering team seeks a Senior Software Engineer specializing in large‑scale, real‑time data streaming systems. It builds high‑performance, fault‑tolerant streaming applications for real‑time analytics, APIs, AI workflows, and mission‑critical data services.

You will architect distributed, event‑driven systems using Java, Kafka, Kubernetes, and modern reactive frameworks. As a senior engineer, you will shape technical direction, mentor engineers, and drive production‑grade reliability across streaming platforms. This role requires deep expertise in distributed systems, concurrency, and cloud‑native microservices.

• Design & Build Real‑Time Streaming Applications
  • Develop high‑throughput, low‑latency streaming applications using Java and Kafka.
  • Design event‑driven microservices that process, enrich, and route real‑time data at scale.
  • Implement reactive, non‑blocking architectures for high concurrency and resilience.
• Architect Scalable Distributed Systems
  • Design and optimize Kafka topics, partitions, consumer groups, and event schemas.
  • Build horizontally scalable services deployed on Kubernetes.
  • Contribute to event‑driven architecture standards and platform design decisions.
• Production Reliability & Performance
  • Optimize performance for throughput, latency, and resource efficiency.
  • Implement observability using metrics, logging, and distributed tracing.
  • Build automated testing strategies for streaming workflows, including integration and load testing.
  • Participate in on‑call rotations and production incident response.
  • Deploy and manage containerized services in Kubernetes environments in GCP or similar cloud environments.
  • Define autoscaling strategies and best practices for resource management.
  • Develop CI/CD pipelines and Infrastructure as Code practices.
• Cross‑Functional Collaboration
  • Partner with Data Engineers to integrate streaming systems with batch pipelines and data platforms.
  • Work with Software Engineers and Product Managers to expose real‑time APIs and services.
  • Collaborate with AI/ML teams to enable real‑time feature pipelines and inference services.
• Technical Leadership
  • Lead architectural reviews and mentor engineers in distributed systems best practices.
  • Drive code quality, documentation, and system design standards.
  • Advocate for scalable, secure, and maintainable engineering solutions.

• Advanced proficiency in Java, including concurrency, multithreading, and JVM performance tuning.
• Strong experience with reactive frameworks such as Spring Web Flux, Project Reactor, or similar.
• Deep understanding of asynchronous, non‑blocking system design.
• Extensive experience with Apache Kafka (producers, consumers, streams, schema registry).
• Strong understanding of partitioning strategies, offset management, rebalancing, and failure recovery.
• Experience designing event schemas and managing schema evolution.
• Familiarity with Kafka Streams, Flink, or similar stream‑processing frameworks.
• Strong hands‑on experience deploying and operating applications in Kubernetes.
• Experience with containerization (Docker) and microservices architecture.
• Knowledge of autoscaling, rolling deployments, and production reliability patterns.

• 5-8 years of experience and strong foundation in distributed systems engineering and event‑driven architecture.
• Proven track record of building and operating large‑scale data streaming systems in production.
• Ability to balance architectural rigor with practical delivery timelines.
• Excellent problem‑solving and collaboration skills.
• Self‑motivated and committed to engineering excellence.

Hiring Salary Range: $ – $. The hiring salary range for this position applies to New York, California, Colorado, Washington state, and most other geographies. Starting pay for the successful applicant depends on a variety of job‑related factors, including but not limited to geographic location, market demands, experience, training, and education. The benefits available for this position include medical, dental, vision, 401(k) plan, life insurance coverage, disability benefits,…