Databricks Architect​/ADMIN

Job Type: Full-Time

Per manager: This isn't an end to end mgmnt, but more of a consultant role. We have an extensive databricks team with a lead already. I need this type of expertise for consultation, pipelines and some automation work...also for forecasting usage.

The Databricks Architect/ADMIN is a senior individual contributor responsible for the design, implementation, and continuous optimization of the enterprise Databricks platform. This role serves as the technical authority for all aspects of the Databricks environment — including workspace governance, Unity Catalog, cluster and compute strategy, data pipeline architecture, and cost management. The Architect works in close partnership with data engineering, analytics, and infrastructure teams, and operates within a broader multi-platform data ecosystem that includes Ab Initio and Fivetran.

A strong background in Unix/Linux systems administration and scripting is essential, as the role requires deep engagement with the underlying compute infrastructure supporting the platform.

• Architect and govern the enterprise Databricks environment, including workspace topology, Unity Catalog structure, and access control frameworks.
• Define and enforce standards for cluster configuration, runtime versions, instance pool utilization, and auto-scaling policies.
• Design scalable, performant data pipeline patterns using Delta Live Tables, Databricks Workflows, and structured streaming.
• Establish architectural standards for Delta Lake — including table formats, partitioning strategies, Z-ordering, and OPTIMIZE/VACUUM scheduling.
• Lead platform integration design with upstream ingestion tools including Fivetran and Ab Initio, ensuring reliable, governed data delivery.

• Administer and troubleshoot Unix/Linux environments underpinning Databricks compute nodes, init scripts, and cluster lifecycle management.
• Develop and maintain shell scripts (Bash) and Python automation for platform operations, monitoring, log aggregation, and maintenance tasks.
• Manage file system operations, permission structures, and data movement tasks in Linux-based storage and compute environments.
• Support EC2/VM-level diagnostics and tuning in coordination with infrastructure and cloud engineering teams.

• Own DBU consumption tracking and reporting; proactively identify optimization opportunities across jobs, interactive clusters, and SQL warehouses.
• Implement and maintain cost attribution models to support chargeback or showback reporting by team, product, or LOB.
• Partner with the Senior Director on capacity planning, contract utilization forecasting, and multi-year commitment management.

• Design and implement data governance frameworks within Unity Catalog, including lineage, tagging, and access auditing.
• Collaborate with Cybersecurity to ensure platform configurations satisfy enterprise security controls, including secrets management, network isolation, and encryption.
• Support audit and compliance activities by maintaining documentation of platform configurations, access policies, and data classification standards.

• Design and implement end-to-end automation frameworks for platform operations, including cluster lifecycle management, job scheduling, alerting, and self-healing workflows.
• Leverage Databricks AutoML, MLflow, and Model Serving capabilities to support the operationalization of machine learning models within the enterprise data platform.
• Integrate AI-assisted development tooling (e.g., Databricks Assistant, Git Hub Copilot) into engineering workflows to accelerate pipeline development and reduce manual effort.
• Identify and drive automation opportunities across ingestion, transformation, data quality, and governance processes — reducing toil and improving platform reliability.
• Collaborate with data science and advanced analytics teams to architect scalable feature engineering pipelines and model deployment patterns on Databricks.
• Evaluate and recommend emerging AI/ML…