Principal Software Engineer - Identity Workflows

Principal Software Engineer

One Identity's One Login team is seeking a Principal Software Engineer to lead the architecture, correctness, and evolution of identity workflow systems across our platform. These systems are responsible for executing identity and resource lifecycle operations at scale and require deep expertise in distributed systems, asynchronous processing, and reliability under real-world conditions.

As a Principal Engineer, you will operate with broad technical ownership—defining standards, guiding implementation, and resolving systemic issues across provisioning workflows, mapping systems, and distributed execution pipelines. You will work across teams to ensure that identity workflows are reliable, scalable, and consistent while evolving legacy implementations toward modern, resilient architectures.

This is a hands-on role that includes system design, debugging complex cross-service workflows, and driving improvements in high-throughput, asynchronous systems. You will also play a key role in mentoring engineers and raising the level of expertise in distributed systems and workflow execution across the organization.

Own the execution of identity and resource lifecycle workflows through scalable, reliable distributed systems that transform intent into consistent, correct outcomes at scale

• Design and evolve distributed workflow systems (e.g., pipelines, queues, workers, transformation layers)
• Ensure correctness of asynchronous execution across retries, failures, and partial completion scenarios
• Define and enforce patterns for idempotency, ordering, and consistency in distributed workflows
• Optimize throughput, parallelism, and system performance across large-scale worker fleets
• Improve reliability through robust retry strategies, back pressure handling, and failure recovery mechanisms
• Define and manage ordering and consistency guarantees in distributed workflows where sequencing and dependencies matter
• Ensure observability and diagnosability of workflows, including tracing, metrics, and runtime inspection
• Ensure deterministic and correct system behavior across retries, reprocessing, and partial execution scenarios
• Identify high-impact areas for improving performance, reliability, and scalability in workflow systems
• Guide evolution of legacy batch or monolithic processing into scalable, distributed execution models

Ideal

Experience:

• Deep experience building and operating distributed systems with asynchronous execution models
• Strong experience with queue-based architectures (e.g., Kafka, SQS, RabbitMQ, or equivalent)
• Experience designing and operating worker-based systems at scale
• Strong understanding of eventual consistency, idempotency, and failure handling in distributed workflows
• Experience with concurrency, parallelism, and multiprocessing across distributed environments
• Experience operating systems in Kubernetes environments, including HPA-driven scaling
• Experience improving throughput, performance, and system efficiency under high load
• Experience debugging complex distributed systems involving queues, workers, and multiple services
• Experience building resilient systems with retry handling, dead-letter queues, and reprocessing strategies
• Experience evolving legacy batch or pipeline systems into modern distributed architectures
• Strong intuition for system behavior under failure, scale, and partial execution
• Experience using AI tools to analyze system behavior, debug workflows, and improve observability

• 8+ years of software engineering experience with ownership of distributed or high-scale production systems.
• Strong backend development experience (Ruby, Node.js, or similar).
• Solid understanding of APIs, distributed architectures, and software design principles.
• Experience designing, building, and operating distributed, asynchronous systems.

• Experience building secure, reliable systems with strong testing practices across unit, integration, and service layers.
• Strong ownership of code quality, validation, and handling complex scenarios.

• Experience building and operating services in AWS or similar cloud environments.
• Good…