Data Engineer​/Enterprise Telemetry & Data Pipeline SME

Position: Data Engineer / Enterprise Telemetry & Data Pipeline SME
San Antonio, Texas

Top Secret w/ SCI

On-site only

Information Technology

** Overview*
* GovCIO is seeking a Senior Data Engineer / Enterprise Telemetry & Data Pipeline SME to design, optimize, automate, and sustain large-scale data collection and streaming pipelines supporting Air Force Intelligence Community missions.

The environment includes a global Elastic deployment with distributed endpoint collectors, syslog and telemetry from Cisco and other network devices, Cribl-based routing and normalization of native and non-native sources, Splunk engineering and integration, and AWS-based infrastructure and services. The primary focus is the movement and optimization of machine-generated data from source to authorized analytics and SIEM destinations.

This is not primarily a data science or dashboard-development role. The successful candidate must be able to evaluate the complete data path, identify collection and transport inefficiencies, engineer scalable solutions, automate repeatable operations, and independently resolve complex ingestion, routing, performance, reliability, and integration problems.  This position will be located on Lackland AFB and will be an onsite position.

** Responsibilities*
* + Serve as the senior technical owner and subject-matter expert for AF IC enterprise telemetry collection, ingestion, transformation, normalization, routing, and delivery architectures.

+ Architect and optimize end-to-end data flows from distributed endpoints, servers, applications, network devices, security platforms, and cloud services into enterprise Elastic, Splunk, and other authorized analytics platforms.

+ Assess collection architectures for bottlenecks, duplicate or unnecessary data, inefficient routing, bandwidth consumption, data-loss risk, processing overhead, and modernization opportunities.

+ Engineer and troubleshoot Cribl Stream and Cribl Edge pipelines, including sources, destinations, routes, transformations, parsing, filtering, enrichment, normalization, buffering, and replay.

+ Engineer Elastic collection and ingestion capabilities using Elastic Agent, Beats, Logstash, ingest pipelines, APIs, data streams, Fleet/integrations, and related technologies.

+ Support Splunk ingestion and integration using forwarders, syslog, HTTP Event Collector, APIs, intermediate collection tiers, and other approved mechanisms.

+ Develop data schemas, field mappings, metadata standards, tagging, enrichment, and normalization practices that improve interoperability and downstream usability.

+ Design resilient pipeline patterns using buffering, queueing, retry, back pressure management, store-and-forward, failover, and recovery for geographically distributed and bandwidth-constrained environments.

+ Establish pipeline observability and data-quality controls to identify dropped, delayed, malformed, duplicated, or incorrectly routed events and measure throughput, latency, queue depth, and error rates.

+ Perform capacity and performance analysis across collectors, transport links, processing tiers, AWS services, and downstream analytics platforms; recommend changes that improve efficiency and scale.

+ Optimize data volume and routing to reduce unnecessary network, compute, storage, cloud, and SIEM ingestion cost without compromising mission or cybersecurity requirements.

+ Design and support AWS-based ingestion, streaming, processing, storage, and monitoring architectures appropriate to mission requirements.

+ Develop Python tooling and automation for data transformation, pipeline validation, API integration, configuration generation, health monitoring, testing, and troubleshooting.

+ Develop and maintain Infrastructure as Code and configuration automation using Terraform, Cloud Formation, Ansible, or equivalent technologies.

+ Manage pipeline code, configuration, and infrastructure artifacts in Git and integrate changes into CI/CD and Dev Sec Ops  workflows with automated validation and controlled promotion.

+ Work with network, cloud, cybersecurity, systems, Elastic, Splunk, and Dev Sec Ops  teams to resolve cross-domain data-flow and infrastructure issues.

+ Ensure data transport and pipeline implementations meet applicable security, access-control, encryption, certificate, auditing, retention, and compliance requirements.

+ Lead root-cause analysis for data loss, latency, parsing, routing, ingestion, scaling, capacity, and integration failures and implement corrective actions that reduce recurrence.

+ Develop architecture and data-flow diagrams,…