Senior Splunk Administrator

Valiant Solutions is seeking a Senior Splunk Engineer to join our rapidly growing and innovative cybersecurity team!

The Splunk Engineer will design, implement, and maintain secure, reliable data ingestion pipelines and integrations across Splunk environments, supporting logs, metrics, events, and other cybersecurity data sources. This role is responsible for Splunk administration, infrastructure performance, data integrity, system upgrades, custom Technical Add-Ons, alerting, and troubleshooting across on-premises and cloud-based environments. The Engineer will collaborate closely with SOC and cross-functional teams to optimize cybersecurity tools, support access controls, resolve technical issues, and improve operational processes and documentation.

The ideal candidate brings strong Splunk engineering and SPL expertise, Linux experience, and the ability to manage high-volume data pipelines while ensuring performance, security, and compliance.

Named one of the Best Places to Work in the Washington DC area for 12 consecutive years
, Valiant is proud of our employee-centric culture and commitment to excellence.

This position allows for 100% remote work. Remote work requires a high level of trust in our employees, and we strictly adhere to the details outlined in our Remote Work Policy below.

• 4 years of Information Technology Experience
• Bachelor of Science in Computer Science, Information Systems, Mathematics, Engineering, related degree or an additional two (2) years of experience.
• 3+ years of Splunk administration or engineering experience.
• Proficiency in configuring and managing Splunk inputs, setting up data ingestion pipelines, and establishing system connections.
• Knowledge of Splunk's Search Processing Language (SPL), data parsing techniques, and the use of regular expressions for data extraction and transformation.
• Skilled in optimizing data pipelines for performance and efficiency, handling large data volumes, and implementing best practices for data integrity and consistency.

• Splunk Core Certified Consultant or Splunk Enterprise Certified Architect certification preferred.
• Splunk Enterprise or Splunk Cloud Certified Admin acceptable.
• Experience as an engineering team lead (representing the team's work to clients).
• Experience working with Splunk, syslog, syslog-ng or systems designed to collect and centralize logs.
• Experience leveraging Cribl to optimize Splunk ingest
• Experience assisting teams with Fraud Analytics
• Strong analytical and problem-solving skills, with the ability to effectively prioritize and execute tasks

• Design, develop and implement processes for ingesting data from various sources into Splunk, ensuring seamless integration and minimal data loss. Processes may include HTTP Event Collector (HEC), Splunk Stream, Splunk Technical Add-Ons (TAs), Universal Forwarder (UF) Collections, custom APIs and other ingest mechanisms.
• Configure and manage data inputs to accommodate different types of data sources, including logs, metrics, and events to determine compliance with M-26-14 requirements at the FISMA system boundary level.
• Establish and maintain secure and reliable connections between Splunk and external systems or data sources.
• Create, monitor, and maintain alerts for failed data inputs.
• Create and maintain custom Technical Add-Ons for non-standard ingest paths leveraging API connections.
• Ensure proper authentication and authorization mechanisms are in place for data transfer and system communication.
• Oversee the configuration and maintenance of Splunk infrastructure, ensuring optimal performance and security of the Splunk environment.
• Collaborate with cross-functional teams to troubleshoot and resolve issues related to Splunk functionality.
• Conduct root cause analysis for incidents and implement preventive measures.
• Utilize Linux skills to manage and maintain the underlying operating system or Cloud based containers of Splunk servers and other security applications.
• Monitor tool health and performance to identify issues, bugs, or potential improvements.
• Develop, review, and update existing operational documentation (SOPs, Job Aids, application checklists, playbooks, etc).
• Support system access controls, including Account Management, Access Enforcement, Information Flow Enforcement, Least Privilege, and workflow for all user account requests and account recertifications.
• Collaborate with the Security Operations Center (SOC) teams for process…