Senior Databricks Platform Engineer; Remote

ICF is seeking a Senior Databricks Platform Engineer to support an independent US Government Agency where you will design, build, secure, and operate governed Azure Databricks Lakehouse capabilities supporting a cloud-based analytics platform, enterprise AI operations, and emerging data initiatives.

Job Location:

Remote

• Design, configure, and operate Azure Databricks work spaces, clusters, jobs, workflows, catalogs, storage credentials, external locations, and service principals for enterprise initiatives.
• Build and optimize batch and real-time ingestion and transformation pipelines using Python, PySpark, SQL, Delta Lake, Delta Live Tables, Auto Loader, and Apache Spark.
• Implement medallion architecture, dimensional models, change data capture, schema evolution, data quality checks, retry and rollback mechanisms, and reproducible processing patterns.
• Configure Unity Catalog governance, including role-based access, lineage, retention, catalog and schema provisioning, and secure integration with ADLS Gen2 and other approved storage.
• Develop frameworks and interfaces that enable AI and analytics teams to access governed training, inference, and reporting data across CDMA, Alpha Rex, and future DMS initiatives.
• Automate Databricks deployments and environment configuration using Terraform, Databricks Asset Bundles, Git Hub Actions, Azure Dev Ops, or comparable tools.
• Integrate Databricks with Azure Data Factory, Kafka, Airflow, APIs, Snowflake, Synapse, and other enterprise data services as required by the solution architecture.
• Monitor pipeline, cluster, and workspace health; diagnose performance and data issues; and use Azure Monitor, Databricks logs, Splunk, or comparable tools to support root cause analysis and operational reliability.
• Support security, ATO, vulnerability remediation, business continuity, disaster recovery, and operational readiness activities for Databricks environments and data products.
• Participate in Agile planning, code reviews, demos, and retrospectives; document architecture, runbooks, data models, governance decisions, and deployment procedures in version-controlled repositories.

• BA/BS/MS in Computer Science, Engineering, Data Science, or equivalent experience, with more than 5 years of data engineering experience and at least 3 years specifically supporting Azure Databricks and related Azure data services.
• Expertise in Apache Spark, Delta Lake, Python, SQL, and Scala, with hands-on experience designing and optimizing large-scale ETL/ELT pipelines.
• Experience with Azure Data Factory, Azure Data Lake Storage, Azure Synapse Analytics, Databricks workflows, data modeling, and big data technologies.
• Experience configuring and managing Databricks work spaces, clusters, compute resources, security controls, access management, and cost optimization.
• Experience implementing CI/CD or infrastructure as code for data platforms using Git and tools such as Terraform, Databricks Asset Bundles, Git Hub Actions, Azure Dev Ops, or comparable technologies.
• Experience with Agile delivery, data governance, data security, performance troubleshooting, and clear communication with technical and business stakeholders.
• Ability to satisfy FDIC personnel eligibility and background-investigation requirements
• U.S. Citizenship required due to federal contract requirements.
• Candidate must reside in the U.S., be authorized to work in the U.S., and all work must be performed in the U.S.

• Experience with Azure Databricks, ADLS Gen2, Unity Catalog, Delta Live Tables, Auto Loader, Azure Data Factory, and Azure Synapse.
• Databricks Certified Data Engineer Associate or Professional certification, Databricks Generative AI Engineer certification, or comparable demonstrated experience.
• Experience with streaming and real-time data using Kafka, Confluent, Structured Streaming, or similar technologies.
• Experience with medallion architecture, CDC, dimensional modeling, Apache Iceberg, Snowflake, Oracle, SQL Server, or other enterprise data platforms.
• Experience supporting Generative AI, NLP, RAG applications, vector stores, agents, or Databricks AI/ML workloads.
• Experience with MLflow, model deployment pipelines, AWS Sage Maker, AWS Bedrock, or cross-cloud data and AI architectures.
• Experience implementing BCDR, high availability, disaster recovery, data retention, and enterprise security controls for cloud data platforms.
• Experience supporting ATO processes, NIST or federal…