Principal Software Engineer

** Overview*
* Microsoft Azure High Performance Computing & AI Engineering (HPC & AI Eng) team is responsible for managing the core platform & fleet of AI High Performance Computing products that customers use to run their most performant and demanding workloads. The AI Customer Experience (AICE) engineering team within the HPC & AI Eng. team is on the front lines managing the flagship supercomputers used by top tier AI customers that enable breakthroughs such as ChatGPT and are highlighted in Top
500, MLPerf and Graph
500 rankings. As a Principal Supercomputing Software Engineer, you will design and develop high volume low latency telemetry pipelines, connect to existing telemetry pipelines, and stitch together data to deliver first to know insights on customer facing issues across the infrastructure stack - from datacenter events to various hardware and networking subsystem events affecting job reliability and causing job interrupts. In this role, you will bring exceptional design and development expertise, with a solid background in large-scale High-Performance Computing & GPU systems, cloud computing platforms, high-performance data processing infrastructure.

This opportunity will give you hands-on experience managing the largest scale of supercomputers delivered to our customers. As a key technical leader, you will engage deeply with strategic customers, directly influencing their business outcomes as well as drive engineering improvements in the Azure ecosystem benefiting the broader fleet. Your work will enable the next wave of growth and innovation in AI and high-performance computing (HPC) in the cloud.
Microsoft's mission is to empower every person and every organization on the planet to achieve more. As employees we come together with a growth mindset, innovate to empower others, and collaborate to realize our shared goals. Each day we build on our values of respect, integrity, and accountability to create a culture of inclusion where everyone can thrive at work and beyond.

** Responsibilities*
* - Architect, design and develop high volume low latency end to end event pipelines that can provide first-to-know-insights on events causing job interrupts and job reliability - Conduct analysis of existing event pipelines to evaluate fidelity, granularity and latency of critical events - Contribute to improving key metrics such as Job Mean Time to Interrupt, Nodes in Service, Mean Time to Resolve on flagship supercomputers by enabling data scientists and domain experts to use the telemetry to identify events & issues at the intersection of datacenter and hardware, develop hypothesis, conduct A/B tests and synthesize results - Partner with cross organizational teams to evaluate available telemetry and latency drive architecture, design, development and deployment of end-to-end solutions to manage core infrastructure including current & next generation datacenter, IT hardware, power & cooling technologies - Drive engineering and operational excellence based on issues and learnings from strategic customers on their usage scenarios to improve product features and capabilities - Partner with teams on continuous learning and continuous improvement programs by leading the resolution of complex incidents, driving root cause analyses and championing initiatives to minimize future customer impact

** Qualifications*
* Required Qualifications:

- Bachelor's Degree in Computer Science or related technical field AND 6+ years technical engineering experience with coding in languages including, but not limited to, C, C++, C#, Java, JavaScript, or Python - OR equivalent experience.

- 5+ years hands on experience designing and developing high volume low latency pipelines using products such as Az Pub Sub , Event Hubs, Azure Stream Analytics, Kafka, Grafana, Event Hubs, Prometheus or equivalent products - 3+ years of experience with one of AI/HPC system management OR High-Speed Networks OR HPC Storage OR managing Cloud Infrastructure   Other

Qualifications:

- Ability to meet Microsoft, customer and/or government security screening requirements are required for this role. These requirements include, but are not…