Data Architect

Job Responsibilities

• Design, provision, and maintain Databricks work spaces, clusters, jobs, and Unity Catalog using Terraform as infrastructure-as-code across multiple environments.
• Develop and own reusable Terraform modules for Databricks, cloud networking, storage, and security, ensuring consistent, repeatable deployments and minimal configuration drift.
• Integrate Databricks Terraform workflows into CI/CD pipelines (Git-based) to automate environment creation, configuration changes, and promotion across dev, test, and prod.
• Implement and enforce Databricks security baselines (cluster policies, access controls, secrets management) via Terraform, in partnership with cloud and security teams.
• Optimize Databricks platform cost and performance by managing cluster types, pools, autoscaling policies, and job scheduling through Terraform-managed configurations.
• Monitor and troubleshoot Databricks infrastructure issues, building observability and alerting around Terraform-driven changes and platform health.
• Collaborate with data engineering and ML teams to codify new Databricks capabilities (feature stores, workflows, libraries) as Terraform resources, enabling self-service and platform scalability.