Observability Engineer

PENNYMAC Pennymac (NYSE: PFSI) is a specialty financial services firm with a comprehensive mortgage platform and integrated business focused on the production and servicing of U.S. mortgage loans and the management of investments related to the U.S. mortgage market. At Pennymac, our people are the foundation of our success and at the heart of our dynamic work culture. Together, we work towards a unified goal of helping millions of Americans achieve aspirations of home ownership through the complete mortgage journey.

Job Overview We're looking for an experienced, forward-thinking Observability Engineer to expand our Observability team in Core Services Engineering, and strengthen our observability capabilities across Pennymac environments. In this role, you will be responsible for designing, implementing, and maintaining our observability platform, with a strong focus on New Relic. You will leverage your expertise in Infrastructure as Code (IaC) to automate and manage our monitoring and alerting infrastructure, ensuring our systems are reliable, performant, and transparent.

You will work closely with Core Services, Dev Ops, Development and Operation teams to foster a culture of proactive monitoring and data-driven decision-making. If you're passionate about automation, cloud-native patterns, and making systems run smarter and safer, we want to hear from you. A Typical Day The Sr Observability Engineer will:
Design & Implement Observability Solutions:
Architect, build, and scale comprehensive monitoring solutions using the New Relic platform, including APM, Infrastructure, Logs, Synthetics, and custom instrumentation (NRQL). Automate with IaC:
Develop, manage, and maintain observability configurations-including alerts, dashboards, and synthetic checks-using Infrastructure as Code (IaC) tools such as Terraform/Open Tofu. Develop Dashboards & Alerts:
Create and refine insightful dashboards and actionable alerting policies in New Relic to provide real-time visibility into infrastructure and application health. Promote Best Practices:
Act as a subject matter expert on observability, guiding teams on best practices for logging, metrics, and tracing to improve system reliability and reduce mean time to resolution (MTTR). Troubleshoot & Optimize:
Analyze performance data and telemetry to identify bottlenecks, troubleshoot production issues, and drive performance optimization efforts across the stack. What You'll Bring Bachelor's degree in Computer Science, Engineering, or a related field (or equivalent experience). 7+ years of experience in a Cloud Engineering role (Observability, Dev Ops, SRE, etc). Proven New Relic Expertise: 3+ years of hands-on experience with the New Relic platform, including deep knowledge of Dashboards, NRQL, APM and setting up effective alerting.

Strong IaC Proficiency: 3+ years of experience managing infrastructure and configurations with IaC tools like Terraform/Open Tofu (preferred), AWS CDK, Cloud Formation, Chef or Ansible. Cloud Platform

Experience:

Extensive hands-on experience working with major cloud providers such as AWS (preferred), GCP, or Azure. Scripting

Skills:

Proficiency in a scripting language such as Python, Go, or Bash for automation and tooling. System Knowledge:
Strong understanding of cloud architecture, networking principles, Windows/Linux Server administration, microservices in a SaaS context, containerization (Docker, Kubernetes), and CI/CD principles. Security & Compliance experience:
Deep understanding of security best practices and their implementation in cloud infrastructure and CI/CD pipelines. Experience working in complexly regulated environments. Excellent problem-solving and troubleshooting skills. Strong communication and collaboration skills. Why You Should Join As one of the top mortgage lenders in the country, Pennymac has helped over 4 million lifetime homeowners achieve and sustain their aspirations of home.

Our vision is to be the most trusted partner for home. Together, 4,000 Pennymac team members across the country are guided by our core values: to be Accountable, Reliable and Ethical in all that we do. Pennymac is committed to conducting a business that makes positive contributions and promotes long-term sustainable growth and to fostering an equitable and inclusive environment, where all employees and customers feel valued, respected and supported.

Benefits That Bring It Home:
Whether you're looking for flexible benefits for today, setting up short-term goals for tomorrow, or planning for long-term success and retirement, Pennymac's benefits have you covered. Some key benefits include:
Comprehensive Medical, Dental, and Vision Paid Time Off Programs including vacation, holidays, illness, and parental leave Wellness Programs, Employee Recognition Programs, and onsite gyms and cafe style dining (select locations) Retirement benefits, life insurance, 401k match, and tuition reimbursement Philanthropy Programs including matching gifts, volunteer grants,…