Senior​/Principal Data Engineer

Sci Tec has been awarded multiple government contracts and is growing our creative Team! Sci Tec, Inc. is a dynamic small business with the mission to deliver advanced sensor data processing technologies and scientific instrumentation capabilities in support of National Security and Defense. We support customers throughout the Department of Defense and U.S. Government in building innovative new tools to deliver unique world-class data exploitation capabilities.

Important Notice:
Sci Tec exclusively works on U.S. government contracts that require U.S. citizenship for all employees. Sci Tec cannot sponsor or assume sponsorship of employee work visas of any type. Further, U.S. citizenship is a requirement to obtain and keep a security clearance. Applicants that do not meet these requirements will not be considered.

We are seeking an experienced Data Engineer to join our Mission Data Processing program. In this role, you will design, build, and maintain scalable ETL pipelines for processing terabyte-scale streaming data and architect databases optimized for machine learning on on-premises hardware using open-source software. The ideal candidate will have expertise in data design patterns such as the Medallion Architecture and data lakehouse technologies to ensure efficient and reliable data processing.

You should be skilled at handling high-throughput, low-latency data ingestion, managing data bursts, and implementing features like time-based partitioning, versioning, auditing, and rollback for historical data replay and event reproducibility. Additionally, you will bring Dev Ops expertise for pipeline automation, Infrastructure as Code (IaC) skills with tools like Terraform and Ansible, and a strong understanding of Dev Sec Ops  practices for maintaining secure and compliant data workflows.

Responsibilities

• Design and optimize ETL pipelines capable of handling high-throughput, low-latency data ingestion, especially during large data bursts
• Implement robust asynchronous processing systems using ZeroMQ to handle large, serialized Protobuf messages
• Create systems that efficiently process sudden, large volumes of data while maintaining performance
• Design strategies for managing back pressure to prevent system overload during high data volumes
• Develop fault-tolerant systems to safeguard data integrity and maintain reliability
• Set up monitoring and alerting mechanisms for proactive response to sudden data load changes
• Build and sustain high-performance databases on on-premises infrastructure, leveraging MinIO or similar object storage solutions for seamless integration with ML workflows
• Apply and manage data design patterns such as the Medallion Architecture to organize data into Bronze, Silver, and Gold layers
• Deploy Delta Lake solutions to combine the flexibility of data lakes with data warehouse performance
• Implement containerization and orchestration solutions using Docker and Kubernetes, and build CI/CD pipelines for automated ETL workflows
• Implement infrastructure provisioning and deployment automation using Terraform and/or Ansible
• Uphold data governance and security protocols to ensure data integrity and compliance with DoD standards, including vulnerability scans and secure configurations
• Lead the evaluation and adoption of open-source technologies that enhance data engineering capabilities
• Work with subcontractors and DoD organizations across sites, accommodating hardware limitations and ensuring seamless integration
• Maintain comprehensive documentation and train teams on best practices and tools in data engineering
• Lead and provide guidance to developers and engineers on architecture, design, and testing decisions
• Provides thought-leadership and subject matter expertise for data engineering and data pipeline orchestration across the company
• Regularly communicate with customers, present status, and engage in program-level meetings and processes
• Other duties as assigned

• Minimum 8 years of experience building and maintaining data pipelines/ETL solutions at scale
• Proficiency in Python, C++, SQL, and RDBMS (Postgre
  
  SQL or similar)
• Experience with object storage (e.g., MinIO), Protocol Buffers, and ZeroMQ
• Fam…