Lead Observability & Data Architect​/Engineer; SC​/DV Cleared

We are hiring for Lead Observability & Data Architect/Engineer (SC/DV Cleared)

Job Type :
Open for Contract (Outside / Inside IR35) & Permanent hiring

Location :
Hybrid - London/Reading/Birmingham/Manchester/Glasgow

• We are seeking an Enterprise Platform Architect or Senior Data Engineer to design and scale national-critical monitoring fabrics and data hubs. Whether your background is rooted in core Observability platforms (Open search/Elastic/Splunk/Open Telemetry) or high-scale Data Engineering (Kafka, Python, Airflow, Open Search/Lucene), your core mission is the same: transforming billions of raw event logs into actionable, secure streaming telemetry.
• Should be able to build high-performance data pipelines to ingest, parse, and structure massive event telemetry across hybrid-cloud and air-gapped environments.
• Manage large-scale cluster deployments, designing optimized indexing, data lifecycle policies, and schema mapping (such as ECS).
• Expertise in Elasticsearch/Open Search, Splunk, or Solr. Strong proficiency in building visualization dashboards (Kibana, Grafana) and managing telemetry collectors (Open Telemetry/OTel, Beats, Prometheus).
• Experience in designing and scaling data pipelines for high-velocity telemetry. Essential expertise in Apache Kafka, Logstash, or streaming/orchestration frameworks (Airflow, Vector, Debezium).
• Hands‑on experience architecting scalable, containerized solutions using Kubernetes (K8s), Docker, and infrastructure‑as‑code (Terraform/Ansible).
• Search & Indexing Engine Depth:
  Prior architecture experience with Open Search, Apache Solr, or Lucene core or Elasticsearch.
• Strong proficiency in Python, Rust, or Bash for automating log parsing, testing, and deployment workflows.
• A strong track record of engineering within highly regulated sectors (eg, MoD, Home Office, BAE, HMRC) with an active UK security clearance (SC/DV).

We highly encourage applications if you bring any of the following equivalent expertise:

• From Big Data/ETL to Telemetry Pipelines:
  If you have built high‑throughput streaming pipelines using Apache Kafka, Airflow, NiFi, or Spark, you already know how to route and transform complex data feeds at scale.
• From Enterprise Search to System Observability:
  If you have engineered core search, indexing, or sharding architectures using Apache Solr, Lucene, or Open Search, you possess the exact foundational knowledge needed to scale an enterprise Elasticsearch cluster.
• From Cloud Infrastructure to Monitoring Fabrics:
  If you are a seasoned Cloud or Infrastructure Architect specializing in secure Kubernetes orchestration, Terraform automation, and air‑gapped system design, you can easily pivot to defining secure logging architectures.