Data Engineering Lead

Position Overview

The Data Engineering Lead is responsible for designing and implementing modern, scalable data architectures to support migration of legacy, file-based analytical systems to AWS Cloud Native environments.

This role leads the transformation of legacy SAS-based data storage models—including flat files, batch outputs, and subsystem-specific data artifacts—into structured, governed, and scalable data models optimized for cloud-native processing.

The Data Engineering Lead will ensure data integrity, performance, and visibility across a system-of-systems modernization initiative, while providing technical leadership for data modeling, ingestion patterns, validation frameworks, and transparency reporting.

Expert-level proficiency in Python and strong experience designing AWS-based data architectures are required.

• Participate in structured system inventory efforts to document:
  • Legacy file-based storage structures
  • SAS dataset dependencies
  • Subsystem data flows
  • Manual gating and handoff processes
• Analyze legacy storage models and design target-state data models aligned to AWS Cloud Native architecture.
• Replace file-driven batch dependencies with:
  • API-based ingestion
  • Event-driven workflows
  • Database-backed storage (e.g., Aurora/Postgres)
• Define canonical data schemas and transformation standards.

• Architect scalable AWS data pipelines using services such as:
  • S3
  • Glue
  • Lambda
  • Event Bridge
  • SNS/SQS
  • Aurora/Postgres
  • Batch
  • Athena
• Design data ingestion, staging, transformation, and validation workflows.
• Establish schema management, versioning, and data lineage practices.
• Optimize data storage for performance, scalability, and cost efficiency.
• Support serverless and containerized data processing architectures.

• Develop advanced Python-based data transformation and validation pipelines.
• Implement modular, reusable data processing components.
• Optimize large-scale data manipulation for distributed execution.
• Develop high-performance ETL/ELT frameworks.
• Embed automated validation checks directly into data pipelines.

• High-volume data processing
• Data validation logic
• Modular data engineering frameworks

• Design and implement automated data validation frameworks to ensure:
  • Functional equivalence during migration
  • Record-level and aggregate-level consistency
  • Downstream compatibility across subsystems
• Develop dashboards and reporting mechanisms providing:
  • Data accuracy metrics
  • Pipeline health indicators
  • Variance detection summaries
• Enable transparency into data transformation impacts across modernization phases.
• Support regression validation through golden datasets and automated comparisons.

• Coordinate with Senior Developers and Requirements Engineers to align data models with application modernization.
• Ensure upstream/downstream data contract stability.
• Prevent data thrashing during phased migration.
• Support orchestration of gated workflows through automated triggers rather than manual file exchanges.
• Collaborate across work streams to establish shared data standards.

• Integrate data pipelines into CI/CD frameworks.
• Support infrastructure-as-code alignment (Terraform/Cloud Formation collaboration).
• Ensure compliance with security controls (IAM, encryption, key management).
• Produce documentation supporting:
  • Architecture review boards
  • Interface control documents
  • Data flow diagrams
• Support ATO-related data validation evidence.

• 8+ years of experience in data engineering or data architecture.
• Expert-level proficiency in Python for data engineering.
• Demonstrated experience transforming legacy file-based systems into cloud-native data architectures.
• Experience developing data models for high-volume, data-intensive applications.
• Deep experience with AWS data services (Glue, Lambda, S3, Aurora/Postgres, Event Bridge, etc.).
• Experience designing scalable ETL/ELT pipelines.
• Experience building analytical dashboards (e.g., Quick Sight or equivalent).
• Experience implementing automated data validation and quality controls.
• Experience working in Agile Scrum Teams.
• U.S. Citizenship required.

• Experience modernizing SAS-based data environments.
• Experience supporting system-of-systems integration programs.
• Experience implementing data lineage and metadata management.
• Experience operating in regulated or federal environments.

• Systems-level thinking across data ecosystems
• Strong schema design and normalization expertise
• Data accuracy and integrity focus
• Automation-first mindset
• Cross-workstream coordination capability

• 401(k) with matching and 100% Vested
• Health Insurance - 3 plans to select from
• Dental insurance
• Vision Insurance
• Health savings account
• Life insurance
• Short Term Disability
• Long Term Disability
• AD&D
• Paid time off
• Professional development assistance
• Training
• Tuition…