Splunk Engineer Job Chicago area,Illinois USA,Business

Job Title:

Splunk Engineer

Job Location:

Chicago, IL Job Type: Contract

Job Description:

Data Orchestration:

Architect the ingestion of the "Three Pillars" (Metrics, Logs, Traces) using Open Telemetry (OTel) collectors.

Aggregation Strategy:

Develop logic to aggregate high-cardinality data to reduce "noise" while maintaining "signal" for troubleshooting.

Analytical Modeling:

Use SPL (Search Processing Language) and Signal Flow to perform pattern analysis, detecting anomalies before they trigger traditional threshold alerts.

Visual Storytelling:

Build executive and technical dashboards that correlate disparate data points (e.g., showing how a spike in 500-errors in Logs relates to a specific span in a Trace
).

Required Hands on Technical Skills
1. Telemetry & Data Specialization Logs:

Proficiency in "Logging-in-Context." You must be able to link logs directly to trace IDs so developers can jump from a failing trace to the specific line of code in the logs.

Metrics:

Expertise in Signal Flow (Splunk's background streaming analytics language). You should know how to calculate percentiles ($P95, P99$), rates of change, and historical averages.

Traces:

Deep understanding of Distributed Tracing
. You must know how to instrument applications (Java, Python, Go) to capture spans and identify bottlenecks in microservices.

2. Pattern Analysis & Aggregation Anomaly Detection:

Ability to configure Metric Finder and MDetector using standard deviations or "Mean Absolute Deviation" to find outliers.

Data Scrubbing:

Skills in using Splunk Ingest Actions or Edge Processors to filter, mask, or aggregate data at the edge to save on license costs and improve search speed.

Pattern Discovery:

Using Splunk's machine learning commands (e.g., find keywords, cluster) to group millions of log events into a few dozen "patterns" for faster root cause analysis.

3. Hands on - Dashboards & Visualization High-Cardinality Handling:

Designing dashboards that don't "break" when viewing thousands of containers.

Contextual Drill-downs:

Building "Glass Tables" (in ITSI) or Unified Dashboards that allow a user to click a metric and immediately see the associated logs.

Frameworks:

Familiarity with the Dashboard Studio and JSON-based dashboard definitions for version control (Git Ops).

Preferred Qualifications & Certifications Dev Ops & IAC skills Splunk Cloud Certified Metrics User:

Focuses on the metrics and alerting side.

Splunk Core Certified Power User:

Essential for mastering complex SPL for log analysis.

Open Telemetry Expert:

Knowledge of the OTel Collector configuration (receivers, processors, exporters) is currently the most "in-demand" skill for this role.