Sr. Data Engineer

Overview

Automation Test Engineers

Halvik Corp delivers a wide range of services to 13 executive agencies and 15 independent agencies. Halvik is a highly successful WOB business with more than 50 prime contracts and 500+ professionals delivering Digital Services, Advanced Analytics, Artificial Intelligence/Machine Learning, Cyber Security and Cutting-Edge Technology across the US Government. Be a part of something special!

The Senior Data Engineer will be a key contributor within the Data Products Delivery Area, responsible for designing, building, and operating scalable, secure, and high-quality data pipelines that power Business Intelligence & Analytics and AIML products. This role supports a Databricks Data Lakehouse using Medallion Architecture, built on Data Fabric principles, and operates within a release-based Agile framework.

This role requires strong hands-on development using Python, PySpark, SQL, and APIs, and experience integrating diverse enterprise and external data sources.

Data Engineering & Architecture

• Design, develop, and optimize end-to-end data pipelines using Python and PySpark.
• Build and maintain data ingestion frameworks leveraging REST APIs, streaming APIs, and batch interfaces.
• Implement Medallion Architecture (Bronze, Silver, Gold) within a Databricks Lakehouse.
• Integrate data from structured, semi-structured, and unstructured sources, including API-based and event-driven sources.
• Apply Data Fabric principles including metadata-driven ingestion, lineage, observability, and reusability.
• Collaborate with Senior Data Architects on logical and physical data models.

Data Products & Delivery

• Deliver data assets as releasable products
• Support downstream consumers including: BI&A dashboards and visualizations
• AIML feature engineering, training, and inference pipelines
• Data services and curated datasets exposed via APIs
• Decompose requirements into well-defined user stories aligned with MoSCoW prioritization.

Data Quality, Governance & Security

• Implement data validation, reconciliation, and quality checks within PySpark pipelines.
• Ensure secure handling of PII and sensitive data, including encryption, masking, and access controls.
• Partner with Cloud Ops (Dev Sec Ops , Fin Ops, Infra Ops) to ensure:
  • Secure API access and secrets management
  • CI/CD automation for Python and PySpark workloads
  • Cost-optimized compute and storage usage

Production Support & Optimization

• Support O&M for production data pipelines, APIs, and analytics products.
• Tune Spark jobs, SQL queries, and API integrations for performance and reliability.
• Implement monitoring, logging, and alerting for data workflows.
• Drive automation and refactoring to improve resiliency and scalability.

Collaboration & Leadership

• Mentor junior and mid-level data engineers on Python, PySpark, and API integration best practices.
• Work closely with:
  • Business Analysts & Data Architects
  • Data Scientists & ML Engineers
  • BI Developers
  • Automation Test Engineers
• Contribute to reusable frameworks, coding standards, and engineering best practices.

Core Technical Skills

• Strong hands-on experience with:
• Python for data engineering and API integration
• PySpark / Apache Spark for large-scale data processing
• SQL (advanced querying and performance tuning)
• Proven experience building API-driven data ingestion pipelines (REST, JSON, OAuth, pagination, throttling).
• Strong experience with Databricks, Delta Lake, and Lakehouse Architecture.
• Experience implementing Medallion Architecture in production environments.

Cloud & Dev Ops

• Experience working in cloud platforms (AWS, Azure, or GCP).
• Familiarity with:
  • CI/CD pipelines for Python and Spark workloads
  • Infrastructure as Code concepts
  • Secure API authentication and secrets management
  • Cost-conscious engineering mindset (Fin Ops exposure is a plus).

Data Management & Analytics

• Experience supporting BI&A workloads at scale.
• Understanding of data modeling for analytics and reporting.
• Experience enabling data consumption for AI/ML use cases.

Agile & Delivery

• Proven experience working in Agile Scrum / Scrum of Scrums environments.
• Comfortable working with release-based delivery and production support.

Nice to Have

• Experience exposing data via internal or external APIs.
• Familiarity with streaming technologies (Kinesis, Firehose, Kafka, Event Hubs, etc.).
• Experience with data governance, cataloging, and lineage tools.
• Exposure to feature stores or ML data pipelines.

Soft Skills

• Strong analytical and problem-solving skills.
• Excellent communication and collaboration abilities.
• Ownership mindset with a focus on reliability and quality.

Company-supported medical, dental, vision, life, STD, and LTD insurance

Benefits include 11 federal holidays and PTO

Eligible employees may receive performance-based incentives in recognition of individual and/or team achievements.

401(k) with company matching

Flexible Spending Accounts for commuter, medical, and dependent care expenses

Tuition Assistance

Charitable Contribution matching

Halvik Corp is…