Lead Data Engineer - Clearance Required

Overview

LMI is seeking a Lead Data Engineer supporting Special Operations Command (SOCOM) on‑site in Crystal City. The position involves providing big data ETL support on classified data and systems to directly support client mission. Active Top Secret SCI clearance and U.S. citizenship are required. LMI is a new breed of digital solutions provider dedicated to accelerating government impact with innovation and speed, investing in technology and prototypes ahead of need and delivering commercial‑grade platforms and mission‑ready AI to federal agencies at commercial speed.

• Design, develop, and implement scalable data pipelines and ETL processes using Apache Airflow, with a focus on data for AI applications.
• Develop messaging solutions utilizing Kafka to support real‑time data streaming and event‑driven architectures.
• Build and maintain high‑performance data retrieval solutions using Elasticsearch/Open Search.
• Implement and optimize Python‑based data processing solutions.
• Integrate batch and streaming data processing techniques to enhance data availability and accessibility.
• Ensure adherence to security and compliance requirements when working with classified data.
• Work closely with cross‑functional teams to define data strategies and develop technical solutions aligned with mission objectives.
• Deploy and manage cloud‑based infrastructure to support scalable and resilient data solutions, including AWS S3.
• Optimize data storage, retrieval, and processing efficiency, focusing on big data.
• Utilize Trino for distributed SQL query processing.
• Leverage Open Meta Data  for data discovery and governance.
• Implement APIs using FastAPI for streamlined data interaction.
• Ensure proper implementation of PKI’s & ACCM’s.
• Integrate DIAS solutions within the data pipeline framework.

• Bachelor’s degree or higher in data engineering, computer science, data science, or similar.
• At least 5 years of experience in data engineering or related work.
• Expertise in Python with significant on‑the‑job experience.
• Significant experience with big data technology and ETL.
• Experience with Airflow as an ETL workflow technology.
• Advanced proficiency in SQL (especially Postgres and/or Trino) with significant on‑the‑job experience.
• Experience with Kubernetes as an operating environment and Helm as infrastructure management.
• Experience with AWS services and cloud architecture.
• Active Top Secret SCI clearance and U.S. citizenship.

• Experience with Postgres or Trino.
• Experience with Apache Iceberg, Hive, HDFS, or Spark.
• Previous DOD experience.

Mid‑Senior level

Full‑time

Information Technology

Software Development

Referrals increase your chances of interviewing at LMI by 2x.