Senior Observability Engineer; Splunk​/SignalFx

Fairfax, United States | Posted on 10/09/2026

• Platform Architecture
  :
  Architect, deploy, and optimize Splunk Enterprise log management and Splunk Observability Cloud (Signal Fx) environments.
• Data Ingestion & Pipelines
  :
  Build and maintain telemetry data pipelines using the Open Telemetry (OTel) Collector, Signal Fx Smart Agents, and Splunk Universal/Heavy Forwarders.
• Monitoring & Alerting
  :
  Design high-performance dashboards, real-time streaming charts, and dynamic alerting mechanisms using Signal Flow or advanced SPL (Search Processing Language) to eliminate alert fatigue.
• Distributed Tracing & APM
  :
  Instrument cloud-native microservices applications (Java, Python, Go, Node.js) to capture distributed traces, metrics, and logs for end-to-end transaction visibility.
• Collaboration & Coaching
  :
  Partner with SRE and Development teams to establish Service Level Indicators (SLIs) and Service Level Objectives (SLOs), providing training on root-cause analysis using the modern Splunk ecosystem.
• Best Practice
  :
  Maintain comprehensive documentation of system architecture, configurations, and troubleshooting procedures aligned with SDLC (Software Development Life Cycle) best practices.
• Up-to-date Security Posture
  :
  Stay current with emerging trends in cloud security, vulnerability management, threat intelligence, and system hardening to enhance monitoring strategies.

• Experience:
  
  4+ years of dedicated engineering experience in platform
  Observability
  , Site Reliability Engineering (SRE), or Dev Ops.
• Proven experience in solution architecture for large-scale enterprise systems with a focus on observability tools such as
  Splunk Enterprise Security
  or
  Signal Fx (Splunk Observability Cloud).
• Tooling Expertise:
  Proven, hands-on experience with Splunk Cloud/Enterprise and Signal Fx (Splunk Infrastructure Monitoring, Splunk APM, and Splunk Log Observer).
• Open Telemetry:
  Deep familiarity with Open Telemetry standards, including configuring OTel Collectors for distributed tracing and streaming metrics.
• Cloud & Containerization:
  Strong expertise monitoring AWS, Azure, or GCP infrastructure alongside container orchestration platforms like Kubernetes (EKS/ GKE) and Docker.
• Infrastructure as Code (IaC):
  Experience automating monitoring deployments via Terraform, Ansible, or Cloud Formation.
• Strong background in IT infrastructure including cloud platforms (AWS, Azure), virtualization (VMware), operating systems (Linux/Windows), and network fundamentals (DNS, VPNs, firewalls).
• Expertise in scripting languages like Python, Bash shell scripting; familiarity with version control systems such as Git for managing codebases.
• Deep understanding of system design principles including microservices architecture, load balancing, RESTful APIs, and system integration techniques.
• Knowledge of database systems including SQL (MySQL, PostgreSQL), No
  
  SQL databases (MongoDB), and data management best practices.
• Experience with Dev Ops practices such as CI/CD pipelines (Jenkins), container orchestration (Kubernetes), configuration management (GPO), and system hardening for security compliance.
• Strong analytical skills in requirements analysis, business analysis, and technical troubleshooting support across
• Excellent cross-functional communication skills to translate technical concepts into clear documentation for stakeholders at all levels.

• Splunk Enterprise Certified Admin, Splunk Cloud Certified Admin, or official Splunk Observability Cloud certifications.
• Proficiency in scripting or coding languages (Python, Go, or Bash) to write custom automation scripts or Open Telemetry plug-ins.