Principal Engineer – Network Tools and Observability

Principal Engineer – Network Tooling And Observability

Wells Fargo is seeking a Principal Engineer to lead the strategy, architecture, and delivery of the enterprise network tooling stack, with a strong emphasis on observability and Agentic AI. This role ensures the bank's networks – including datacenter, branch, cloud, WAN/SD-WAN, and SASE/edge – are instrumented with robust, scalable, and secure tooling to support operational excellence, compliance, and cost efficiency. The engineer will drive integration of telemetry, topology, and analytics platforms to accelerate incident resolution, enable data-driven planning, and support Secure by Design operations.

This position sits within the Innovation and Transformation team in Secure Network Services, part of the Cyber Security organization.

About this role:

We are hiring a Principal Engineer – Network Tooling and Observability within the Cyber Security organization's Innovation and Transformation team. In this role, you will lead the evolution of the bank's network tooling ecosystem, ensuring it delivers actionable insights, automation, and resilience across hybrid environments.

Key Responsibilities:

• Strategy &
  
  Roadmap:
  
  Advise leadership on multi-year strategy for network tools and observability (NPMD, telemetry pipelines, topology, synthetic monitoring), aligned with SRE, Security, and Cloud initiatives.
• Architecture:
  Design and evolve the observability stack, selecting tools and technologies (e.g., SNMP, flow/IPFIX, syslog, streaming telemetry/gNMI, Open Telemetry, APM/NPM, log analytics, time-series DB) for scalability across on-prem, cloud, and branch environments.
• Delivery Leadership:
  Oversee implementation and integration with ITSM/CMDB, CI/CD, configuration compliance, and ITIL-based incident/problem/change workflows.
• Cross-Functional Collaboration:
  
  Partner with Net Ops, SRE, Sec Ops, Cloud, and Lines of Business to translate requirements into architecture and measurable outcomes.
• Blueprints & Standards
• Define baselines for telemetry and data collection:
• SNMP polling and SNMP traps
• Syslog ingestion and correlation
• Flow data (Net Flow, sFlow, IPFIX, etc.)
• Packet capture tools for deep traffic analysis
• Integration with vendor-based controllers and SaaS platforms
• Define tooling standards across platforms such as:
• Spectrum, DX Net Ops, NFA
• Big Switch / DMF, CVP / CVaaS / DCNM
• DNAC / CatC, Net Scout / Giga Mon
• Prime / Airwave / CV-Cue
• Enterprise Impact:
• Lead resolution of complex challenges requiring deep evaluation across multiple domains.
• Translate strategic business objectives and enterprise technology landscape into engineering solutions.
• Provide vision and technical direction to leadership for innovative, large-scale solutions.
• Operational Excellence:
• Optimize processes and drive robust automation.
• Identify inefficiencies and promote continuous improvement.
• Leadership & Mentorship:
• Align stakeholders through clear technical communication.
• Mentor teams and deliver knowledge transfers to upskill SMEs.
• Present technical concepts to senior leaders.

Required Qualifications:

• 7+ years of Engineering experience, or equivalent demonstrated through one or a combination of the following: work experience, training, military experience, education
• 7+ years in large‑scale network engineering/operations, with 5+ years focused on tooling and observability.

Desired

Qualifications:

• Expert knowledge of enterprise networks (L2/L3, BGP/EVPN, DC fabrics, SD‑WAN, SASE, firewalls, LB, DNS/DHCP/IPAM) and cloud networking (Azure/AWS/GCP: VNet/VPC, Transit, Direct Connect/Express Route).
• Deep experience with telemetry & tooling: SNMP, Net Flow/IPFIX/sFlow, syslog, streaming telemetry (gNMI/NETCONF/RESTCONF), synthetic monitoring, path analytics, and topology.
• Hands-on experience with tools such as Thousand Eyes, App Dynamics, Splunk/Elastic, Prometheus/Grafana, Service Now.
• Automation and integration skills:
  Python, Ansible, Terraform, CI/CD, REST APIs, webhooks, event pipelines.
• Strong grasp of SRE principles (SLIs/SLOs/error budgets), reliability and resilience (RTO/RPO, DR, chaos engineering).
• 5+ years of technical design experience.
• 5+ years of network or network security design experience.
• Strong…