Senior Databricks Platform Engineer

AWS | Unity Catalog | Lakeflow | Terraform | Python / SQL

Radiant | Remote

Annual base salary: $205,000–$275,000

Citizenship: U.S. citizen or lawful permanent resident (green card holder)

Background investigation: IRS Minimum Background Investigation (MBI) / Moderate-risk Public Trust

About Radiant

Radiant is a fast-growing small business delivering digital services to public-sector clients. We specialize in end-to-end data engineering and AI-driven software development, helping government partners modernize critical systems and better serve their communities.

About the Role

We’re hiring a Senior Databricks Platform Engineer to design, automate, and operate the Databricks Lakehouse Platform supporting IRS
. As part of the Data Modernization & Governance team, you’ll build the secure, reliable foundation for data engineering, reporting, analytics, and AI/ML capabilities that support operational decision-making.

This is a platform ownership role and not solely a pipeline-development position. You’ll shape architecture, write production automation, implement governance controls, and improve performance, reliability, and cost efficiency. You’ll work closely with architecture, security, governance, and engineering teams to make the platform easier to operate and easier for others to use effectively.

What You’ll Do

• Build and operate the platform. Design, configure, and maintain Databricks environments on AWS, including work spaces, compute policies, clusters, SQL warehouses, and serverless capabilities where supported and authorized.
• Make security and governance operational. Implement Unity Catalog catalogs, schemas, storage access, and least-privilege permissions for groups and service principals. Align identity integration, networking, encryption, auditing, and access controls with AO requirements.
• Automate infrastructure and delivery. Use Terraform, Python, Databricks APIs, and Declarative Automation Bundles—formerly Databricks Asset Bundles—to provision resources and deploy workloads through version-controlled CI/CD pipelines with testing, environment promotion, and rollback procedures.
• Develop reusable data-engineering patterns. Build and optimize ingestion frameworks, ETL/ELT pipelines, and orchestration using Apache Spark, Delta Lake, Lakeflow Spark Declarative Pipelines—formerly Delta Live Tables—and Lakeflow Jobs—formerly Databricks Workflows. Support integrations with enterprise data sources, BI platforms, and AI/ML environments.
• Improve performance and cost efficiency. Diagnose Spark and SQL bottlenecks, test scalability, and optimize data layouts and compute configurations. Apply right-sizing, autoscaling, auto-termination, and workload isolation; track Databricks Unit (DBU) consumption and associated cloud costs to support forecasting and capacity planning.
• Strengthen production reliability. Establish actionable monitoring, logging, alerts, and operational metrics. Troubleshoot incidents, lead root-cause analysis, manage runtime upgrades and releases, and implement and test recovery procedures. Support security audits and authorization-to-operate activities.
• Help other engineers deliver effectively. Establish reusable templates, reference architectures, engineering standards, and practical runbooks. Mentor data engineers and platform users, explain technical trade-offs, and evaluate new capabilities against program needs.

What You’ll Bring

• Hands-on production Databricks experience on AWS
  , including platform administration, Unity Catalog, compute management, SQL warehouses, and workflow orchestration—not just notebook development.
• Strong Python, SQL, Apache Spark/PySpark, and Bash skills, with demonstrated experience troubleshooting distributed processing and improving workload performance.
• Experie…