Senior Databricks Engineer

Koniag Emerging Technologies is looking for a highly qualified, experienced and self-motivated individual to perform the duties of a Senior Databricks Engineer to support our client located in Washington, DC. This position requires an Active Secret clearance.
This is a remote position.

Benefits include medical, dental, and vision insurance, 401(k) retirement plan, paid time off, paid parental leave, life and disability insurance, flexible spending accounts, commuter benefits and tuition reimbursement.

The Senior Databricks Engineer will be responsible for designing, developing, and deploying scalable data infrastructure while driving key data engineering initiatives. By leveraging the Databricks platform and deep expertise in modern data warehousing and the Medallion Architecture, the engineer will build data pipelines supporting the Department of State’s “4

Ps” data lake. The ideal candidate will be able to support the following functions:

• Cross-Functional Collaboration:
  
  Work closely with project managers, team members, and business owners to gather requirements, communicate progress, and ensure data meets customer needs.
• Architecture & Design:
  Should be capable of architecting and implementing scalable, highly available data pipelines using Databricks, adhering strictly to the Medallion Architecture (Bronze, Silver, Gold layers) to ensure data quality, consistency, and optimized reporting.
• Data Quality Governance & Monitoring:
  Establish comprehensive data quality standards and implement monitoring dashboards to enable business owners and responsible parties to track and proactively address data issues.
• Pipeline Development:
  Write clean, efficient, and production-ready code using PySpark and SQL for complex data extraction, transformation, and loading (ETL/ELT) processes. Expertise in DLT and structured streaming pipelines.
• API Integration:
  Design and build robust data ingestion pipelines to reliably pull from various third-party external systems using RESTful API integrations.
• Automated Deployments (CI/CD):
  Implement infrastructure and pipelines as code. Utilize Databricks Asset Bundles (DABs) integrated with Azure Dev Ops to manage and automate deployments across Development, Staging, and Production environments safely and reliably.
• Code Reusability:
  Develop, package, and maintain custom Python libraries and modules to establish common frameworks used across multiple data projects and teams.
• Data Warehousing:
  Apply foundational and advanced data warehousing concepts (e.g., dimensional modeling, slowly changing dimensions) to optimize data storage and retrieval for downstream analytics and business intelligence.
• Performance Tuning:
  Monitor, troubleshoot, and optimize Databricks clusters and Spark jobs to ensure maximum efficiency and cost-effectiveness.
• External BI & Native Dashboarding:
  Securely sharing data with external BI tools (such as PowerBI or SAP Business Objects) and developing dashboards and capabilities natively within the Databricks environment.
• Mentorship & Leadership:
  Serve as a technical SME for the data engineering team, conducting code reviews, establishing best practices, and mentoring other staff members.

• Active SECRET clearance
• Knowledge of Object Oriented Programming (OOP) concepts and proficiency in one or more programming or scripting languages (Java, C#, Python, JavaScript, Power Shell)
• Experience:
  
  8+ years of dedicated Data Engineering experience, with at least 3+ years functioning in a senior capacity focused on the Databricks ecosystem.
• Core
  
  Languages:
  
  Expert-level proficiency in PySpark and complex SQL.
• Python Library Management:
  Proven ability to create, package, and manage custom Python libraries (e.g., building Wheel files) for…