Observability Engineer; Splunk

Position: Observability Engineer (Splunk)
As a leading financial services and healthcare technology company based on revenue, SS&C is headquartered in Windsor, Connecticut, and has 27,000+ employees in 35 countries. Some 20,000 financial services and healthcare organizations, from the world's largest companies to small and mid-market firms, rely on SS&C for expertise, scale, and technology.

Job Description

Job Title:

Observability Engineer (Splunk)

Locations:
Jacksonville, FL;
Boston, MA;
Kanas City, MO | Hybrid 6x a month

Get To Know Us:

As a leading financial services and healthcare technology company based on revenue, SS&C is headquartered in Windsor, Connecticut, and has 27,000+ employees in 35 countries. Some 20,000 financial services and healthcare organizations, from the world's largest companies to small and mid-market firms, rely on SS&C for expertise, scale, and technology.

Why You Will Love It Here!

* Flexibility:
Hybrid Work Model and Business Casual Dress Code, including jeans

* Your Future: 401k Matching Program, Professional Development Reimbursement

* Work/Life Balance:
Flexible Personal/Vacation Time Off, Sick Leave, Paid Holidays

* Your Wellbeing:
Medical, Dental, Vision, Employee Assistance Program, Parental Leave

* Wide Ranging Perspectives:
Committed to Celebrating the Variety of Backgrounds, Talents and Experiences of Our Employees

* Training:
Hands-On, Team-Customized, including SS&C University

* Extra Perks:
Discounts on fitness clubs, travel and more!

What You Will Get To Do:

Splunk Platform (On‑Prem)

* Assist in the day-to-day health, reliability, and performance of Splunk Enterprise (on‑prem), including installs, upgrades, monitoring, backups, and recovery.

* Design, operate, and evolve distributed Splunk architectures, including Search Head Clusters (SHC) and Indexer Clusters.

* Manage ingestion and indexing pipelines end-to-end, including forwarders, parsing rules, and index design/retention.

* Develop and maintain SPL searches, dashboards, alerts, and data models that are operationally meaningful and actionable.

* Operate storage and retention strategies; contribute to Smart Store planning/implementation where appropriate.

Observability Enablement

* Partner with infrastructure and application teams to onboard telemetry correctly and sustainably (logs, metrics, traces, and events).

* Define and enforce observability standards (naming, tagging, retention, alert thresholds, and dashboard patterns).

* Improve signal-to-noise: reduce alert fatigue and increase actionable, well-contextualized alerts.

Linux & Infrastructure Operations

* Execute production changes independently and safely: instance resizing/replacement, storage migrations, network/IP cutovers, and maintenance-window execution.

* Apply sound engineering judgment to minimize downtime and preserve rollback options.

* Create and improve runbooks, automation, and post-change verification checklists.

Required Qualifications

Splunk (On‑Prem Splunk Enterprise)

* Hands-on experience administering Splunk Enterprise in on‑prem environments (not exclusively Splunk Cloud).

* Strong understanding of configuration layering, knowledge objects, and Splunk's data pipeline (input → parsing → indexing → search).

* Experience with distributed Splunk (SHC and indexer clustering), including troubleshooting and performance tuning.

* Working knowledge of SPL, dashboards, and alerting suitable for production operations.

Linux / CLI Fluency (Non‑Negotiable)

* Comfortable operating primarily from the command line; can inspect and manipulate files and system state quickly and safely.

* Bash, systemd, process troubleshooting, and /proc familiarity.

* Text and data tooling: vi (or equivalent), regex, sed/awk, jq, pipelines.

* SSH identities & certificates, rsync, RPM packages/repos, tar/zip, DNS basics, curl.

* File system and storage competence: mounts, disk space management, LVM concepts, growth/resize patterns.

* Practical Git skills (beyond the happy path).

Operational Expectations

* Plan changes with impact analysis, communications, validation, and rollback.

* Demonstrate production rigor: backups, change windows, and post-change verification.

* Maintain clear documentation and handoffs; leave systems better…