Data Infrastructure Engineer

Job Overview

Title: Data Infrastructure Engineer

We are seeking a Data Infrastructure Engineer to build and operate the data platform that powers AI/ML analytics modules. This role is ideal for an engineer who enjoys platform building, automation, and enabling advanced analytics through trusted, well‑governed data.

• Design and implement batch and streaming ingestion from APIs, relational databases, file drops, event streams, and external partners.
• Build and optimize ETL/ELT pipelines to produce curated, analytics‑ready datasets for reporting and ML consumption.
• Implement incremental processing patterns, change data capture (CDC) approaches, and data contract standards.

• Build and manage a scalable lakehouse on AWS object storage (e.g., S3) using open table/file formats and delta/lakehouse concepts such as ACID tables, schema evolution, and time travel.
• Optimize performance and cost through partitioning, compaction, lifecycle policies, and efficient compute/storage usage.
• Establish environment standards for dev/test/prod and consistent promotion across stages.

• Implement a managed metadata repository for cataloging, ownership, glossary/definitions, tagging, and discoverability.
• Enable end‑to‑end lineage to support auditability and impact analysis.
• Implement governance controls including policy‑based access, data classification, retention, and secure data handling.
• Build operational data quality checks (freshness, completeness, validity, anomaly detection) and publish SLAs/SLOs.

• Implement automated cloud provisioning in AWS using IaC for consistent environments and secure-by-default baselines.
• Build and enhance CI/CD for data pipelines, including automated tests, validation gates, promotion workflows, and rollback strategies.
• Improve observability with metrics, logs, alerts, dashboards, runbooks, and incident response readiness.

• Work closely with engineering, security, networking, and application teams to support mission needs and delivery timelines.
• Maintain high‑quality engineering documentation, including SOPs, system diagrams, and secure configuration baselines.
• Summarize and present findings and recommendations—both written and verbal—to technical and non‑technical stakeholders.

• Data pipeline, data ingestion, lakehouse, AWS, SQL, metadata, data engineering, CI/CD, Databricks, git, Terraform, Jenkins, AWS Code Pipeline

• Must obtain and maintain a Federal or DoD “PUBLIC TRUST” before onboarding. Candidates with an active PUBLIC TRUST or suitability are preferred.
• Bachelor’s degree in Engineering, IT, Computer Science, or related field (or equivalent experience).
• Minimum of four (4) years experience building production data pipelines and/or data platforms.
• Strong experience implementing data ingestion and ETL/ELT workflows, including data modeling and transformation best practices.
• Hands‑on experience building a data lake/delta lake on AWS using object storage and modern table formats/patterns.
• Proficiency in SQL and one programming language commonly used for data engineering (Python preferred; Scala/Java acceptable).
• Experience with metadata management and governance: cataloging, lineage, ownership, access controls, classification, and policy enforcement.
• Experience implementing automated AWS provisioning using IaC and operating across multiple environments.
• Experience building or operating CI/CD pipelines for data workflows (testing, packaging, deployment automation, environment promotion).
• Solid security fundamentals: IAM/least privilege, encryption, secrets management, secure SDLC practices.

• Hands‑on experience with Databricks.
• Hands‑on experience utilizing modern Dev Ops practices, including tools like Git, Terraform, Jenkins, AWS Code Pipeline, and Docker.
• Experience utilizing AI‑assisted coding tools (e.g., Git Hub Copilot, ChatGPT, Cursor, Kiro) to safely accelerate implementation while maintaining strict code quality through testing,…