Sr. Data Engineer

Halvik Corp delivers a wide range of services to 13 executive agencies and 15 independent agencies. Halvik is a highly successful WOB business with more than 50 prime contracts and 500+ professionals delivering Digital Services, Advanced Analytics, Artificial Intelligence/Machine Learning, Cyber Security and Cutting-Edge Technology across the US Government. Be a part of something special!

The Senior Data Engineer will be a key contributor within the Data Products Delivery Area, responsible for designing, building, and operating scalable, secure, and high-quality data pipelines that power Business Intelligence & Analytics and AIML products. This role supports a Databricks Data Lakehouse using Medallion Architecture, built on Data Fabric principles, and operates within a release-based Agile framework.

This role requires strong hands‑on development using Python, PySpark, SQL, and APIs, and experience integrating diverse enterprise and external data sources.

• Design, develop, and optimize end‑to‑end data pipelines using Python and PySpark.
• Build and maintain data ingestion frameworks leveraging REST APIs, streaming APIs, and batch interfaces.
• Implement Medallion Architecture (Bronze, Silver, Gold) within a Databricks Lakehouse.
• Integrate data from structured, semi‑structured, and unstructured sources, including API‑based and event‑driven sources.
• Apply Data Fabric principles including metadata‑driven ingestion, lineage, observability, and reusability.
• Collaborate with Senior Data Architects on logical and physical data models.

• Deliver data assets as releasable products
• Support downstream consumers including: BI&A dashboards and visualizations, AIML feature engineering, training, and inference pipelines, Data services and curated datasets exposed via APIs
• Decompose requirements into well‑defined user stories aligned with MoSCoW prioritization.

• Implement data validation, reconciliation, and quality checks within PySpark pipelines.
• Ensure secure handling of PII and sensitive data, including encryption, masking, and access controls.
• Partner with Cloud Ops (Dev Sec Ops , Fin Ops, Infra Ops) to ensure:
  • Secure API access and secrets management
  • CI/CD automation for Python and PySpark workloads
  • Cost‑optimized compute and storage usage

• Support O&M for production data pipelines, APIs, and analytics products.
• Tune Spark jobs, SQL queries, and API integrations for performance and reliability.
• Implement monitoring, logging, and alerting for data workflows.
• Drive automation and refactoring to improve resiliency and scalability.

• Mentor junior and mid‑level data engineers on Python, PySpark, and API integration best practices.
• Work closely with Business Analysts & Data Architects, Data Scientists & ML Engineers, BI Developers, Automation Test Engineers
• Contribute to reusable frameworks, coding standards, and engineering best practices.

• Strong hands‑on experience with Python for data engineering and API integration, PySpark / Apache Spark for large‑scale data processing, SQL (advanced querying and performance tuning)
• Proven experience building API‑driven data ingestion pipelines (REST, JSON, OAuth, pagination, throttling).
• Strong experience with Databricks, Delta Lake, and Lakehouse Architecture.
• Experience implementing Medallion Architecture in production environments.

• Experience working in cloud platforms (AWS, Azure, or GCP).
• Familiarity with CI/CD pipelines for Python and Spark workloads, Infrastructure as Code concepts, Secure API authentication and secrets management
• Cost‑conscious engineering mindset (Fin Ops exposure is a plus).

• Experience supporting BI&A workloads at scale.
• Understanding of data modeling for analytics and reporting.
• Experience enabling data consumption for AI/ML use cases.

• Proven experience working in Agile Scrum / Scrum of Scrums environments.
• Comfortable working with release‑based delivery and production support.

• Experience…