Kafka Engineer Pleasanton, CA Onsite

Kafka Engineer

Location:

Pleasanton, CA (onsite)

• Design and operationalize a multi-environment Confluent Cloud footprint (Prod/Non-prod) including RBAC, Service Accounts, API Keys, Private Networking (VPC peering/Private Link), IP allow lists, and encryption.
• Define topic architecture standards (naming, partitions, replication, retention, compaction), consumer group strategy, and exactly-once/at-least-once semantics.
• Implement Schema Registry and Stream Governance (compatibility modes, lifecycle, breaking change controls).
• Stand up connectivity via Confluent Cloud Connectors
• Establish Terraform/IaC modules for repeatable provisioning (environments, clusters, topics, ACLs, connectors, schemas).

• Partner with Security to implement least-privilege, RBAC, secrets management, and compliance (PII, PCI/SOX/GxP as applicable).
• Define and enforce data governance (ownership, contracts, lineage, retention, DLP).
• Build SLOs/SLAs, error budgets, and incident response runbooks; coordinate with SRE for DR, multi-zone resilience, and capacity planning.

• Instrument metrics, logs, traces across Confluent Cloud and clients (lag, consumer health, throughput, latency, error rates, dead letter queues).
• Integrate platform telemetry with Datadog/Prometheus/Grafana/New Relic/Splunk ; set alerts and automated remediation where feasible.
• Manage upgrades, quotas, cost optimization, and platform hygiene (ACL audits, orphaned topics/connectors cleanup).

• Partner with app teams to design event contracts, idempotent producers, retry & DLQ patterns, and back pressure handling.
• Provide reference implementations and client libraries/configs (Java,.NET, Python, Node) with secure defaults and resiliency patterns.
• Run an “ onboarding desk ”: intake, discovery, architecture review, capacity estimates, provisioning via Terraform pipelines, and go-live checklists.
• Conduct training, documentation, and office hours ; champion best practices for event-driven architectures.