Principal Infrastructure Engineer

Surescripts serves the nation through simpler, trusted health intelligence sharing, in order to increase patient safety, lower costs and ensure quality care. We deliver insights at critical points of care for better decisions - from streamlining prior authorizations to delivering comprehensive medication histories to facilitating messages between providers.

The Principal Infrastructure Engineer is crucial to ensuring the stability, efficiency and security of the products we operate as crucial components of the nation's healthcare ecosystem. As a member of our Operations team, a Principal Infrastructure Engineer is expected to have knowledge and experience spanning many disciplines with the ability to lead, communicate and execute as a subject matter expert on specific topics or systems.

This role may be called upon to troubleshoot deeply technical problems, architect a solution for a new cloud-native application or write an Ansible module to support deployment of new security tooling. At the Principal level, our Platform Engineers are expected to be able to speak on a broad variety of topics and successfully lead projects both internal to our Systems or Operations teams as well as in collaboration with diverse groups of technical and non-technical team members.

This position has a specific focus on systems performance tuning and analysis that is becoming even more crucial as we begin operations in Google Cloud Platform. Sizing, tuning troubleshooting and performance monitoring now draw a direct line to the budget as we transition to the minute-by-minute pricing models of the cloud. Knowing where we can gain efficiency and where we cannot sacrifice performance has become a crucial part of our architectural discussion and ongoing operational health.

• Participate in development of strategic goals and plans for new and existing systems infrastructure.
• Research, design, architect, and implement new technologies and solutions that span on-premise and cloud platforms in a method consistent with our security policies and infrastructure-as-code principals.
• Implement performance tuning strategies to optimize system performance, resource utilization, and overall efficiency.
• Collaborate with other teams to design and implement scalable and resilient infrastructure solutions.
• Assist in support of issues impacting the performance or reliability of production transactions.
• Recommend/implement policies, procedures, and technologies to ensure system security through secure system access, monitoring and control.
• Participate as necessary in routine security evaluations.
• Implement automation solutions for system provisioning, configuration management, and performance tuning.
• Ensure compliance with policies, standards, licensing agreements, configuration guidelines, and best-practices for maintaining regulatory compliance.
• Ensure all systems and processes are well-documented and comply with internal policies and industry standards.

• Bachelor's degree in IT, computer science, information sciences or related field or equivalent experience
• 8+ years of experience in systems engineering role
• Experience in a production operation environment with high uptime requirements.
• Familiarity with configuration management tools (Ansible preferred).
• Knowledge of virtualization and containerization technologies - Vmware (vSphere / ESXi 8x) and Kubernetes preferred
• Proficiency in scripting languages (Bash, Powershell, Python, Perl).
• Experience with performance, logging and monitoring tools (Zenoss, Splunk, Datadog, Prometheus, Grafana, etc.)
• Strong background in system metrics analysis and monitoring.
• Experience working in cloud environments (GCP preferred)

• Experience operating in an environment with sensitive health data
• Experience with Red Hat, RockyOS and Windows Certifications like RHCE or Google Cloud Architect/Engineer
• Understanding of network operations, network troubleshooting and the ability to understand the output of packet captures
• Experience as a Team Lead, or non-people manager leadership role.

Surescripts embraces flexibility…