Cribl Engineer

Alexandria, Virginia

Secret

Fully remote

IT Infrastructure & Network Engineering & Operations

** Overview*
* GovCIO is currently hiring for Data Warehousing Specialist (Network Engineer) of Infrastructure Operations This position will be located within the United States and will be fully remote.

** Responsibilities*
* + Develop apply best practices and tools for data ingestion, indexing, and management to optimize data sources and refine data collection processes to capture only pertinent data.

+ Plan and perform Cribl platform upgrades (Leader, Worker, and Edge nodes) following defined change control procedures.

+ Manage and optimize the Cribl distributed infrastructure, ensuring scalability, stability, and efficient data routing.

+ Continuously monitor Cribl performance, including throughput, queue depth, and worker health metrics.

+ Develop and maintain Cribl pipelines for new data sources, implementing filtering, sampling, and enrichment logic.

+ Migrate existing Splunk forwarder-based data inputs to Cribl for improved control and flexibility.

+ Build and maintain Cribl Packs for standardized configurations across multiple environments.

+ Implement data reduction and enhancement workflows to minimize ingestion volume and improve data quality.

+ Maintain and enhance Ansible playbooks for automated deployments, configurations, and upgrades.

+ Integrate Git Ops CI/CD pipelines (e.g., Git Lab, Jenkins, Terraform) to manage configuration-as-code for both Splunk and Cribl.

+ Develop, test, and review merge requests related to dashboards, alerts, saved searches, and data onboarding pipelines.

+ Perform Splunk core upgrades (indexers, search heads, cluster masters, deployers) ensuring backward compatibility and minimal downtime.

+ Upgrade and validate Splunk Add-ons and Apps, maintaining functionality and CIM compliance.

+ Develop and maintain custom props, transforms, event types, and lookups to normalize data consistently.

+ Ensure CIM compliance for all add-ons and source types used across the platform.

+ Handle escalations from Operations and perform deep-dive troubleshooting on ingestion, parsing, or performance issues.

+ Perform break/fix analysis on Splunk core services such as KVStore, clustering, deployment server, and scheduler.

+ Conduct performance tuning for search optimization, bucket management, and scheduler balancing across SHC.

+ Design and maintain retention, archival, and index management strategies to align with business and compliance goals.

+ Manage license allocation, volume forecasting, and capacity planning across indexer clusters.

+ Develop and maintain monitoring and alerting integrations for Cribl and Splunk infrastructure health.

+ Collaborate with Operations on incident triage, root cause analysis, and postmortem documentation.

+ Create and maintain runbooks and engineering guides for deployments, upgrades, and troubleshooting.

+ Participate in architecture and design discussions to ensure Splunk and Cribl meet enterprise scaling and reliability needs.

+ Implement security and compliance controls including token rotation, TLS configurations, and secret management via Vault or GCP Secret Manager.

+ Perform disaster recovery testing and validate replication and failover processes across clusters.

+ Collaborate with governance teams to align on data retention, anonymization, and privacy requirements.

+ Support continuous improvement by analyzing ingestion efficiency, performance benchmarks, and automation opportunities.

+ Lead knowledge-sharing sessions and technical handoffs with Operations for newly deployed features or pipelines.

** Qualifications*
* Bachelor's with 13+ years (or commensurate experience) OR Masters Degree or higher (in a related discipline) with 10 years

Required Skills and Experience

+ Skills in data warehousing and specifically processing of computer generated log data.

*
* Clearance Required:

** Must be able to obtain and maintain AOUSC Public Trust

Preferred Skills and Experience

+ Extensive experience with Cribl data engine

** Posted Salary Range*
* USD $ - USD $ /Yr.

** Company Overview*
* GovCIO is a team of transformers--people who are passionate about transforming government IT. Every…