Lead Data Engineer - Time Data Processing

Choosing Capgemini means choosing a company where you will be empowered to shape your career in the way you’d like, where you’ll be supported and inspired by a collaborative community of colleagues around the world, and where you’ll be able to reimagine what’s possible. Join us and help the world’s leading organizations unlock the value of technology and build a more sustainable, more inclusive world.

We're seeking a seasoned Lead Software Engineer to architect, build, and scale real‑time data processing platforms that power event‑driven applications and analytics. You'll lead the design of streaming microservices, govern data quality and lineage, and mentor engineers while partnering with product, platform and security stakeholders to deliver resilient, low‑latency systems.

• Own design & delivery of high throughput, low latency streaming solutions using technologies such as Confluent Kafka, Apache Flink, Hazelcast, Kafka Streams, Kafka Connect, and Schema Registry.
• Design and implement microservices and event‑driven systems with robust ETL/ELT pipelines for real‑time ingestion, enrichment and delivery.
• Establish distributed caching and in‑memory data grid patterns (e.g., Redis, Hazelcast) to optimize read/write performance and session/state management.
• Define and ope rationalise event gateways / event grids for event routing, fan‑out and reliable delivery.
• Lead data governance initiatives – standards for metadata, lineage, classifications, retention, access controls and compliance (PII/PCI/SOX/GDPR as applicable).
• Drive CI/CD best practices (pipelines, automated testing, progressive delivery) to enable safe, frequent releases; champion Dev Sec Ops  and "shift left" testing.
• Set SLOs/SLAs, track observability (tracing, metrics, logs) and optimise performance at scale (throughput, back pressure, state, checkpointing).
• Work with Security, Platform and Cloud teams on networking, IAM, secrets, certificates and cost optimisation.
• Mentor engineers, conduct design reviews and enforce coding standards and reliability patterns.
• Guide platform and delivery roadmap.

• 10+ years in software engineering; 5+ years designing large‑scale real‑time or event‑driven platforms.
• Expert with Confluent Kafka (brokers, partitions, consumer groups, Schema Registry, Kafka Connect), Flink (Data Stream/Table API, stateful ops, checkpointing), Hazelcast and/or Kafka Streams.
• Strong in ETL/ELT design, streaming joins/windows, exactly‑once semantics and idempotent processing.
• Experience with microservices (Java/Python), REST/gRPC, protobuf/Avro and contract‑first development.
• Hands‑on with distributed caching and in‑memory data grids; performance tuning and eviction strategies.
• Cloud experience in Azure, AWS or GCP; containers, Docker and Kubernetes.
• Experience in production‑grade CI/CD (Jenkins, Bamboo, Harness or similar), Infrastructure as Code (Terraform/Helm).
• Robust observability (Prometheus/Grafana/Open Telemetry, Splunk/ELK or similar) and resilience patterns (circuit breakers, retries, DLQs).
• Practical data governance: metadata catalogs, lineage, encryption, RBAC.
• Excellent communication; ability to lead design, influence stakeholders and guide cross‑functional delivery.
• Core competencies:
  
  Architectural Thinking, Systems Design, Operational Excellence, Security & Compliance, Team Leadership, Stakeholder Management.

• Experience with CDC, Kafka Connect custom connectors, Flink SQL, Beam.
• Streaming ML or feature store integration (online/offline consistency).
• Multi‑region/disaster‑recovery for streaming platforms.
• Experience with zero‑downtime migrations, blue/green and canary deployments.

• Flexible work.
• Healthcare including dental, vision, mental health and well‑being programs.
• Financial well‑being programs such as 401(k) and Employee Share Ownership Plan.
• Paid time off and paid holidays.
• Paid parental leave.
• Family‑building benefits like adoption assistance, surrogacy and cryopreservation.
• Social well‑being benefits such as subsidised backup child/elder care and tutoring.
• Mentoring, coaching and learning programmes.
• Employee Resource Groups.
• Disaster Relief.