Principal Engineer, Core Infrastructure

Kilt Platform Engineer

As part of the TSO/AOCI organization, our mission is to build Kilt, the platform on which OCI services are built, deployed, and operated across every region. We are investing in core platform capabilities that let service teams focus on their product rather than on the undifferentiated work of running it at scale, and we are doing it with a small group of engineers concentrated on high-leverage, low-level systems.

Within Kilt, the Observability & Supervision team owns how the platform sees itself and reacts. We build the systems that collect metrics and logs from every host and service, evaluate that signal against what healthy is supposed to look like, escalate to the right owner with the right context, and increasingly act on their own, whether that means scaling capacity, triggering recovery, or adjusting behavior based on the live state of upstream dependencies.

This is real-time distributed systems work with strict latency budgets, where a decision has to be both fast and correct under partial information, because a wrong action taken across a fleet is worse than no action at all.

For this role, we are looking for engineers with a passion for understanding how systems fail, not only how they work. You should be comfortable designing distributed control loops, reasoning about performance and correctness at scale, and owning a component from design through production operation. You should value simplicity, be direct in technical discussion, and be excited by problems where the answer is not yet known.

We are a small, growing team with broad ownership and real autonomy. You will work with motivated, diverse people who care about doing things well, and you will have the support to do your best work in a workplace where you belong.

Key Responsibilities

• System Design & Architecture
  - System Scalability:
• Lead the development and implementation, and begin to architect, components of scalable distributed systems that support horizontal and vertical scaling to meet system demands, including leveraging distributed state management tools.
• Optimize code and/or systems for large-scale data processing and high-throughput requirements to support hyper-scale systems.
• Define scalability requirements for owned components and ensure design and implementation requirements are met.
• Design systems to scale with elasticity (e.g., effectively scaling both up and down).
• Leverage data plane platforms to effectively handle large-scale data retrieval, storage, and processing.
• Design performance and load testing.
• System Design & Architecture
  - System Reliability Design:
• Build and design fault-tolerant components and systems capable of withstanding in-service updates by implementing redundancy, replication, and automatic failover mechanisms.
• Design systems to effectively handle service disruptions (e.g., network partitions) by prioritizing consistency, availability, or partition tolerance.
• Implement and optimize approaches to handle network unreliability, including load-shedding, throttling, and rate-limiting.
• Design components and systems that are durable and adhere to service level objectives (SLOs), setting expectations for availability and durability of other computing services within the department.
• System Design & Architecture
  - System Reliability Performance:
• Define key performance indicators (KPIs) and telemetry to identify gaps or issues in running systems.
• Build and customize moderately complex dashboards, telemetry systems, and alerting mechanisms to proactively monitor components and system health.
• System Design & Architecture
  - Correctness / Availability:
• Design and implement functional and correctness requirements for feature sets and/or systems in new or existing systems.
• Design complex test scenarios (e.g., fault-injection, brown-out) to evaluate system correctness.
• Implement data replication and synchronization techniques to maintain data integrity and availability.
• Operational Troubleshooting & Incident Management:
• Take a proactive role in diagnosing, debugging, and resolving issues in active components and systems to support ongoing operation, and mentor others in these processes.
• Implement strategies to prevent interruptions, ensuring no maintenance windows are required for customers and users when resolving issues.
• Maintain expertise in owned components and systems to ensure effective troubleshooting and performance.
• Meet operational readiness expectations through design and implementation.
• Serve in operational support rotations, providing guidance in incident response and root cause investigations.
• Compliance & Security:
• Implement robust security measures to protect data and applications in multi-tenant environments, including encryption techniques and access controls.
• Execute remediation plans to address identified security gaps.
• Ensure cloud infrastructure is in compliance with industry standards and regulations and that documentation is up to date.
• Automation & Change Management:
• Develop and maintain…