Databricks Consultant (ID#5473

New Value Solutions, a national IT consulting company, is seeking a Databricks Consultant to design, lead implementation, and operationalize a secure, scalable Databricks foundation, with a strong focus on Unity Catalog, data governance, platform security, workspace architecture, storage integration, identity and access management, and production-ready data engineering patterns.

The successful candidate will work with data engineering, cloud infrastructure, security, governance, and business teams to establish an enterprise-grade Databricks environment that supports analytics, machine learning, data products, and governed self-service access.

Unity Catalog is Databricks’ unified governance layer for data and AI assets, providing centralized access control, lineage, discovery, auditing, and governance capabilities across Databricks work spaces

Databricks Foundation Implementation

• Design and lead implementation of the core Databricks platform foundation across development, test, and production environments.
• Define workspace architecture, naming standards, environment separation, identity model, network design, and workspace lifecycle management.
• Configure Databricks account-level settings, work spaces, groups, service principals, credentials, access policies, and administrative controls.
• Establish best practices for cluster policies, SQL warehouses, jobs, workflows, repos, secrets, and compute governance.
• Lead Implementation of platform standards for scalability, cost control, monitoring, operational support, and production readiness.

Unity Catalog Design and Implementation

• Design and deploy Unity Catalog as the enterprise governance layer for Databricks.
• Define the Unity Catalog metastore strategy, catalog, and schema structure, naming conventions, ownership model, and environment separation.
• Configure catalogs, schemas, tables, views, volumes, external locations, storage credentials, and permissions.
• Implement fine-grained access control using Unity Catalog privileges, groups, service principals, row filters, column masks, and attribute-based access controls where applicable.
• Establish governed data access patterns for data engineers, analysts, data scientists, and business users.
• Enable data discovery, lineage, auditability, and data asset management using Unity Catalog capabilities. Unity Catalog supports access control, discovery, lineage, auditing, governed tags, data classification, quality monitoring, data sharing, and AI governance

Cloud Storage and Security Integration

• Design secure integration between Databricks and cloud storage platforms such as ADLS Gen2, AWS S3, or Google Cloud Storage.
• Configure storage credentials, external locations, managed storage, and secure access patterns.
• Work with cloud infrastructure and cybersecurity teams to implement networking, private endpoints, encryption, firewall rules, secrets management, and least-privilege access.
• Ensure alignment with enterprise security, privacy, compliance, and data governance requirements.
• Define recommended patterns for Delta Lake, medallion architecture, streaming, batch processing, and data product development.
• Support migration of existing tables, data pipelines, notebooks, and workflows to Unity Catalog-enabled architecture.
• Advise teams on managed versus external tables, external volumes, schema design, permissions, and lifecycle management.
• Create reusable templates, standards, and implementation guides for Databricks development teams.

Governance, Operations, and Documentation

• Document Databricks platform architecture, Unity Catalog design, security model, operational procedures, and support processes.
• Develop runbooks for workspace administration, access provisioning, incident response, job monitoring, and cost management.
• Define roles and responsibilities for platform administrators, data owners, data stewards, engineers, and consumers.
• Support knowledge transfer and training for internal teams.
• Provide recommendations for platform maturity, automation, CI/CD, observability, and ongoing governance.

• Strong hands-on experience implementing and administering Databricks in an enterprise environment.
• Proven…