Databricks Solutions Expert; Remote

Position: Databricks Solutions Expert (Remote)
United States

Suitability/Public Trust

Fully remote

Information Technology

** Overview*
* GovCIO is hiring a Databricks Solutions Expert to support the Department of Veterans Affairs (VA) Data Modernization initiative who will architect, design, and implement Azure Databricks as a multi-tenant, governed Lakehouse platform that can be adopted by thousands of analytics teams across the enterprise. This role blends architecture (platform, security, governance, networking) with engineering (data ingestion, transformation, streaming, ML lifecycle) to deliver a secure, scalable, cost‑efficient analytics foundation on Azure.

This position is fully remote within the United States.

** Responsibilities*
* The Databricks Solutions Expert will will define platform patterns, build reference implementations, set standards (cluster policies, Unity Catalog governance, CI/CD), and drive enablement so product, analytics, and data science teams can deliver reliable, compliant, and performant insights at scale.

** Platform Architecture & Design*
* + Blueprint the Azure Databricks landing zone: workspace topology (prod/non-prod), network architecture (VNet injection, Private Link, NAT), secure connectivity to ADLS Gen2, Azure SQL/MI, Event Hubs, and other data sources.

+ Governance with Unity Catalog: multi-region metastores, catalog/schema/table design, data classification tiers, row/column-level security, data lineage, and cross-domain data sharing patterns (Delta Sharing).

+ Lakehouse foundations:
Delta Lake storage design (bronze/silver/gold), medallion data flow standards, partitioning, Z‑ordering, OPTIMIZE/VACUUM policies, and performance best practices (Photon, AQE).

+ Scalability for "thousands of teams": multi-workspace strategy, tenancy model (shared vs. dedicated), workspace baselines, cluster policy tiers, and guardrails to prevent noisy-neighbor and cost runaways.

+ Reliability: HA/DR strategy, regional deployments, backup/restore, versioning, and repeatable environment provisioning via Terraform.

** Security, Compliance & Access Control*
* + Identity & access:
Entra  (Azure AD) SSO, SCIM user/group provisioning, service principals/managed identities, attribute-based/role-based access controls mapped to Unity Catalog.

+ Secrets & credentials:
Key Vault-backed secret scopes, credential passthrough patterns, token hygiene (PAT governance).

+ Data protection: encryption at rest/in transit, private endpoints, data exfil and egress controls, policy-as-code, and audit logging to Log Analytics or secure storage.

+ Compliance: enforce enterprise policies (PII/PHI handling), data retention, legal hold, and regulatory reporting with auditable lineage.

** Engineering & Enablement*
* + Pipelines: build DLT (Delta Live Tables) pipelines and jobs for batch and streaming (Structured Streaming) with CDC (e.g., via Auto Loader) from enterprise sources.

+ Performance engineering: optimize notebooks/SQL/ETL (Photon, caching, skew mitigation), tune cluster sizing, and set standards for reliable, fast jobs.

+ MLOps: integrate MLflow for experiment tracking, model registry, feature store (Unity Catalog), and serving patterns where appropriate.

+ Observability: end-to-end monitoring (jobs, clusters, UC audits), dashboarding, alerting, and SLOs; integrate with Azure Monitor/Log Analytics.

+ Enablement: build reusable reference accelerators (templates, example notebooks, data products), run playbooks, and conduct office hours to uplevel 1000s of teams.

** Dev Ops, Automation & Cost Management*
* + Infrastructure-as-Code: provision work spaces, catalogs, cluster policies, and permissions via Terraform (Databricks provider), with pipelines in Azure Dev Ops/Git Hub.

+ CI/CD: notebook/package deployment, testing harnesses (dbx/pytest),…