Principal Engineer - Resiliency

Fore Flight is redefining the future of aviation technology - equipping pilots, operators, and decision-makers with the software tools to forge bold new digital paths that push past the expected. We stand with those who dream bigger, fly farther, and seek to explore the next frontier in the cockpit and operations center. The role is preferred as Hybrid working in Austin TX, Houston TX or Denver CO, we will consider virtual for the right candidate.

• Own the resiliency strategy and reference architecture for the AWS estate: set availability targets by service tier, define the standard patterns for multi-AZ and multi-region operation, and drive existing services toward them
• Own the decision framework for active-active versus active-passive versus warm-standby and pilot-light topologies, including the real tradeoffs in cost, operational complexity, and achievable RTO/RPO, and the migration path for services that need to move between them
• Lead business continuity and disaster recovery end to end: RTO/RPO target-setting, automated failover design, regular DR exercises and game days, recovery runbook ownership, and translating exercise findings into durable architectural change
• Run AWS Well-Architected Reviews across service teams with emphasis on the Reliability and Operational Excellence pillars, and convert findings into a prioritized, tracked remediation backlog rather than a report that sits on a shelf
• Drive resiliency into the container platform: ECS/Fargate and EKS workload placement, capacity headroom, failure-domain isolation, graceful degradation, and recovery behavior under partial-region impairment
• Drive resiliency into the data tier so stateful workloads recover as predictably as stateless ones, covering Aurora and RDS multi-AZ and Global Database topologies, DynamoDB global tables, Elasti Cache, and S3 cross-region replication, with clear expectations for failover time, replica lag, and data-loss boundaries
• Encode resilience in automation:
  IaC-defined failover paths, self-healing and auto-remediation, drift detection, and continuous fault injection so recovery capability is exercised routinely instead of documented once
• Own the reliability signal set for core infrastructure: SLOs and error budgets, health-check and failover-trigger design, dependency mapping, and blast-radius analysis across control-plane and data-plane signals
• Raise the quality bar on incident response for availability events, lead resiliency-focused postmortems, and convert repeat failure modes into architectural fixes and platform guardrails
• Operate effectively across a complex organizational context, translating continuity and availability requirements from engineering, product, and business stakeholders into concrete AWS design and validated recovery capability

• 12+ years of engineering experience, with at least 7 as the primary architect or technical owner of infrastructure resiliency and high-availability architecture at scale
• Deep hands-on AWS production experience in large multi-account, multi-region environments, including VPC and networking design, Route 53, ALB/NLB, Global Accelerator, and the IAM and account-boundary decisions that constrain failover
• Demonstrated ownership of highly available architectures where multi-AZ is the baseline and multi-region is real for critical workloads, with credible judgment about when active-active is warranted and when active-passive or warm-standby is the better tradeoff
• Track record owning BC/DR strategy end to end: setting RTO/RPO targets, designing and automating regional failover, and running genuine recovery exercises rather than tabletop-only reviews
• Expert command of the AWS Well-Architected Framework, particularly the Reliability pillar, with experience leading formal reviews and driving the resulting remediation to completion
• Production depth with container platforms on AWS (ECS/Fargate, EKS, or both): scaling behavior, placement and failure domains, deployment safety, and degradation characteristics under failure
• Strong data-tier resiliency experience with Aurora or RDS (multi-AZ, Global Database, failover behavior, replica lag) and at least one of DynamoDB global tables, Elasti Cache replication, or S3 cross-region replication
• Deep production experience designing and operating IaC at scale, Terraform preferred (CDK acceptable), with strong opinions on module strategy, state management, and guardrail enforcement across many accounts and environments
• Expert command…