Sr Data Engineer

Silo Smashers, Inc. is a leading management and IT consulting firm, serving as a trusted advisor and solutions partner to a diverse range of government, business, and not-for-profit organizations. Applying proven expertise and methodologies, we help our customers align and mobilize every aspect of their operations, strengths, and resources to achieve Peak Performance and increased business value today and through the years ahead.

Silo Smashers is hiring for a Senior Data Engineer to support a government customer. This position will be responsible for architecting, developing, optimizing, and maintaining high‑performance data ingestion, transformation, streaming, and analytical pipelines within Databricks‑enabled Lakehouse environments. The selected candidate will serve as a senior technical expert for Databricks Lakehouse engineering, advanced Delta Lake optimization, governed data design (Unity Catalog), medallion‑architecture enforcement, structured streaming, and platform‑integrated cloud data engineering across enterprise platforms.

This role is expected to collaborate closely with the Project Manager / Contractor Lead, federal technical teams, and cross‑task platform, SRE, governance, privacy, and development personnel.

• Serve as the principal technical authority for Databricks data engineering, pipeline optimization, streaming, and Lakehouse standards across programs.
• Lead design reviews, technical roadmaps, and architectural decisions in partnership with federal SMEs and platform/security teams.
• Build and optimize batch and streaming pipelines using PySpark, Lakeflow Spark Declarative Pipelines, Delta Lake, Auto Loader, and Workflows with robust error handling, retries, alerting, and monitoring.
• Lead and enforce medallion‑architecture ingestion and refinement patterns, data lifecycle management, and governed data storage tiering.
• Implement advanced performance tuning and cost‑efficiency strategies, including cluster/sizing optimization, serverless compute adoption, workload health monitoring, and resource governance.
• Provide root cause analysis, disruption/outage reporting, and production support for Databricks jobs, pipelines, applications, and platform features.
• Coordinate with Privacy & Governance teams to enforce Unity Catalog RBAC/ABAC, row/column‑level security, governed tags, lineage, and compliance controls.
• Mentor mid‑level/junior engineers, perform code reviews, and uphold high engineering quality through testing strategies, quality gates, reproducible builds, and Dev Sec Ops  best practices.

• Demonstrated expertise designing and implementing Databricks Lakehouse solutions, including:
  • Delta Lake table design, schema evolution, expectations, and medallion‑architecture (bronze/silver/gold).
  • Databricks Workflows, Jobs, Auto Loader, COPY INTO, Structured Streaming, Databricks SQL Warehouses, and Unity Catalog governance models.
  • Performance and cost optimization (Photon, file sizing, Z‑ORDER, clustering/partitioning, caching, AQE, shuffle/skew mitigation).
• Hands‑on experience building distributed data pipelines using Python, PySpark, and cloud‑native services across AWS and/or Microsoft Azure.
• Demonstrated experience administering and optimizing relational and No
  
  SQL data systems (e.g., PostgreSQL, Aurora, DynamoDB) and designing efficient, secure schemas.
• Experience developing and maintaining CI/CD pipelines using Git‑based workflows (e.g., Git Hub Actions, Azure Dev Ops), Databricks Repos, Asset/Bundles, Terraform/IaC, and automated testing frameworks.
• Experience supporting federal environments and compliance‑aligned development.
• Experience applying agile development methodologies, including backlog management, sprint planning, and…