Senior Data Engineer

Chantilly, Virginia, United States;
Herndon, Virginia, United States

We are seeking a Senior Data Engineer to lead the development and optimization of data pipelines, data movement tools, and data preparation services to support advanced analytics and enterprise integration. This role requires deep expertise in handling both structured and unstructured data, designing scalable data architectures, and developing reliable ETL solutions. The ideal candidate has significant experience with both cloud-native and on-premise technologies, and thrives in collaborative, mission-driven environments.

Clearance Requirement: TS/SCI with Polygraph

Design, implement, and maintain scalable data pipelines using tools such as Spark, Apache Iceberg, Trino, Open Search, EMR, NiFi, and Kubernetes

Perform data ingestion, extraction, formatting, and transformation for a wide range of data types and formats

Clean, preprocess, and format data for exploration, analysis, and operational use

Develop ETL tools and scripts and enhance existing pipelines using coding best practices

Work with Data Scientists, analysts, and mission partners to align data pipelines with mission goals

Collaborate with software engineers to configure and maintain data services and back-end components

Ensure data quality, standardization, and provenance through collaboration with testing and data quality teams

Produce comprehensive documentation including ETL mappings, data lineage, code usage guides, and access instructions

Participate in enterprise working groups to define and promote data standards across systems

Support both one-time and ongoing data extraction and transformation efforts across various repositories

10+ years of experience in data lifecycle engineering

Deep expertise in ETL development, pipeline design, and data architecture

Strong coding skills in Python, SQL, Spark, and other data engineering tools

Experience with COTS and open-source tools including Elastic Search and Apache Ni Fi

Strong understanding of cloud and on-premise data storage and processing solutions

Experience working in Agile development environments

Proven track record of integrating diverse data sources into enterprise systems

Experience deploying solutions on cloud platforms (e.g., AWS EMR, S3, RDS, EKS)

Familiarity with data governance, data lineage tracking, and security requirements

Hands-on experience with container orchestration using Kubernetes

Knowledge of data lake architectures and streaming data processing

Excellent communication skills and experience collaborating with cross-functional teams

Languages & Frameworks:
Python, SQL, Spark

Dev Ops & CI/CD:
Git, Agile methodologies, automated testing frameworks

Tackle large-scale data engineering challenges with national impact

Work with cutting-edge cloud and open-source data tools

Collaborate with a multidisciplinary team of engineers and analysts

Flexible, mission-driven environment with opportunities for leadership

M1 Technology is an equal opportunity employer and values diversity. We do not discriminate in hiring on the basis of race, color, religion, sex, national origin, age, disability, veteran status, or any other characteristic protected by federal, state, or local law.