Principal Core Infrastructure Engineer

** Job Description*
* + Leads development and begins architecting components of scalable, elastic distributed systems.

+ Defines and enforces scalability requirements for owned components; optimizes code and data paths for high‑throughput, hyper‑scale workloads; and leverages data plane platforms for large‑scale retrieval, storage, and processing.

+ Designs fault‑tolerant, in‑service‑upgradable systems using redundancy, replication, failover, and policies for partitions, applying load‑shedding, throttling, and rate‑limiting to handle network unreliability while meeting SLOs.

+ Establishes KPIs and telemetry; builds proactive dashboards and alerts; and designs complex validation (fault injection, brownouts), replication, and synchronization for correctness and durability.

+ Proactively diagnoses and resolves production issues, mentors peers, and ensures operational readiness.

+ Implements robust security controls, executes remediation, maintains compliance documentation, and develops IaC and automation that enable safe patching, updates, and rollbacks within change‑management plans.

** Responsibilities*
* Key Responsibilities

System Design & Architecture - System Scalability:

-Lead the development and implementation, and begin to architect, components of scalable distributed systems that support horizontal and vertical scaling to meet system demands, including leveraging distributed state management tools.

-Optimize code and/or systems for large-scale data processing and high-throughput requirements to support hyper-scale systems.

-Define scalability requirements for owned components and ensure design and implementation requirements are met.

-Design systems to scale with elasticity (e.g., effectively scaling both up and down).

-Leverage data plane platforms to effectively handle large-scale data retrieval, storage, and processing.

-Design performance and load testing.

System Design & Architecture - System Reliability Design:

-Build and design fault-tolerant components and systems capable of withstanding in-service updates by implementing redundancy, replication, and automatic failover mechanisms.

-Design systems to effectively handle service disruptions (e.g., network partitions) by prioritizing consistency, availability, or partition tolerance.

-Implement and optimize approaches to handle network unreliability, including load-shedding, throttling, and rate-limiting.

-Design components and systems that are durable and adhere to service level objectives (SLOs), setting expectations for availability and durability of other computing services within the department.

System Design & Architecture - System Reliability Performance:

-Define key performance indicators (KPIs) and telemetry to identify gaps or issues in running systems.

-Build and customize moderately complex dashboards, telemetry systems, and alerting mechanisms to proactively monitor components and system health..

System Design & Architecture - Correctness / Availability:

-Design and implement functional and correctness requirements for feature sets and/or systems in new or existing systems.

-Design complex test scenarios (e.g., fault-injection, brown-out) to evaluate system correctness.

-Implement data replication and synchronization techniques to maintain data integrity and availability.

Operational Troubleshooting & Incident Management:

-Take a proactive role in diagnosing, debugging, and resolving issues in active components and systems to support ongoing operation, and mentor others in these processes.

-Implement strategies to prevent interruptions, ensuring no maintenance windows are required for customers and users when resolving issues.

-Maintain expertise in owned components and systems to ensure effective troubleshooting and performance.

-Meet…