AI Systems Engineer - Data & State Management - Senior

Location:

Anywhere in Country

At EY, we’re all in to shape your future with confidence.

We’ll help you succeed in a globally connected powerhouse of diverse teams and take your career wherever you want it to go. Join EY and help to build a better working world.

We are seeking an AI Systems Engineer to own the stateful backbone of EY’s AI-native platform, including the data stores, memory tiers, and event streaming systems that hold and move every piece of state in EY’s Hybrid AI Multi‑Environment Runtime (HAI). This role ensures that agentic AI workloads have durable, performant, and consistent access to data across cloud, on‑prem, edge, client‑managed, and air‑gapped environments.

Within HAI, this is a distinct discipline from platform operations and model serving. This role owns everything that must remember, persist, or flow: relational and key‑value state, vector and graph stores, object storage, durable workflows, and the streaming and change‑data‑capture pipelines that connect them. It is the layer that makes the platform stateful, reliable, and event‑driven.

It is ideal for a data‑infrastructure engineer who is equally comfortable operating production databases and high‑throughput streaming systems, who treats data durability, consistency, and recoverability as non‑negotiable in regulated client contexts, and who understands that state is the hardest part of any distributed platform to get right.

• Own the memory and data stores: relational and durable state (PostgreSQL, DBOS durable workflows), caching (Redis/Valkey), vector stores (Qdrant/Milvus/PGVector), knowledge graphs (Neo4j), and object/block storage (MinIO, OpenEBS Mayastor), across every environment and tenant.
• Own event streaming and async messaging:
  Apache Kafka (Strimzi), NATS Jet Stream (agent-to-agent), Debezium (change data capture), Apache Flink (stream processing), and Apicurio/Cloud Events (schema and event contracts).
• Own data durability, consistency, and recoverability: replication, backup/restore, point-in-time recovery, and cross-environment data movement, tiered by RPO/RTO.
• Build and operate streaming and CDC pipelines that move data reliably between stores and services, with schema governance and evolution that prevents breaking changes across producers and consumers.
• Make state multi-tenant and portable, ensuring isolation, performance, and consistent semantics whether running on managed cloud services or self-hosted OSS in an air‑gapped environment.
• Provide the data and lineage substrate that downstream governance, observability, and AI knowledge capabilities depend on, as well as integrating with lineage tooling.

• Deep expertise operating production databases and data stores at scale, including relational, key‑value, vector, graph, and object storage.
• Strong command of streaming and event-driven architectures (Kafka, NATS, CDC, stream processing) and the consistency tradeoffs they involve.
• A durability-first mindset: thinking in terms of consistency, recoverability, blast radius, and data correctness under failure.
• Ability to operate stateful systems consistently across managed cloud and self-hosted OSS in cloud, on‑prem, edge, and air‑gapped environments.
• Strong grasp of schema governance and evolution, preventing breaking changes across producers and consumers.
• Strong communicator able to guide consuming teams toward the right storage and streaming patterns.
• Orientation toward reliability and toil reduction through automation and infrastructure-as-code for data systems.

• Bachelor’s or Master’s degree in Computer Science or related technical field.
• 8+ years operating production data infrastructure, streaming systems, or database platforms at scale.
• Hands‑on expertise with relational databases (PostgreSQL) and caching (Redis/Valkey), including HA, replication, and backup/recovery.
• Exposure to AI/ML data patterns — embeddings, retrieval, feature/state stores for agentic workloads.
• Production experience with vector and/or graph databases (Qdrant, Milvus, PGVector, Neo4j) in AI/ML contexts.
• Deep experience with event streaming and…