Power Platform Architect Job San Diego area,California USA,IT/Tech

Role:
Platform Architect with Databricks

Location:

United States – Remote Mode:
Contract

Experience:

10+yrs Responsibilities:

Design & Deployment:
Architect and oversee the deployment of Azure Databricks work spaces, including VNet Injection, secure cluster connectivity (no public IP), and private endpoints.
Configuration Management:
Define and manage Databricks cluster policies, instance types, Databricks Runtime (D ) versions, and auto-scaling configurations.
Performance Tuning:
Monitor and optimize Databricks workspace and job performance, advising on cluster sizing, Spark configurations, and Delta Lake optimizations.
Capacity Planning:
Forecast and plan for future Databricks resource needs based on platform usage trends.
Cost Optimization:
Identify and implement strategies to optimize Azure Databricks costs (e.g., proper cluster sizing, auto-termination, spot instances, reserved instances).
Troubleshooting:
Provide expert-level troubleshooting for complex Databricks issues, spanning cluster failures, job errors, performance bottlenecks, and connectivity problems.

2. Data Governance & Unity Catalog:

Unity Catalog Design:
Architect and implement the Unity Catalog metastore, including the setup of catalogs, schemas, volumes, and external locations.
Fine-Grained Access Control (FGAC):
Define and enforce data access policies within Unity Catalog, securing tables, views, and managed objects at row and column levels where required.
Data Lineage & Audit:
Leverage Unity Catalog capabilities for data lineage tracking and support auditing requirements for data access.
Managed Tables & Volumes:
Advise and govern the usage of managed and external tables/volumes within Unity Catalog.

3. Security & Access Management (Entra , SCIM, Key Vault, ACLs):

* Azure Entra n:
Design and implement seamless integration of Azure Databricks with Azure Entra (formerly Azure Active Directory) for Single Sign-On (SSO) and user/group synchronization.
SCIM (System for Cross-domain Identity Management):
Configure and manage SCIM provisioning for automated user and group lifecycle management between Entra Databricks.
Azure Key Vault Integration:
Architect and implement secure secret management for Databricks using Azure Key Vault-backed secret scopes for notebooks, jobs, and applications. Advise on credential passthrough configurations.
Access Control Lists (ACLs):
Define and enforce ACLs on Azure Data Lake Storage Gen2 (ADLS Gen2) for Databricks service principals and user access. Implement table-level, view-level, and object-level ACLs within Unity Catalog and non-Unity Catalog environments.
Compliance & Auditing:
Ensure Databricks configurations meet organizational security and compliance standards. Assist in auditing access logs and activity.

4. Networking & Connectivity:

Secure Network Design:
Architect and implement secure network connectivity for Azure Databricks, including VNet peering, private endpoints, and network security groups (NSGs).
Firewall Rules:
Define and manage firewall rules to control inbound and outbound traffic for Databricks work spaces, ensuring secure communication with other Azure services and on-premises resources.
DNS & Routing:
Ensure proper DNS resolution and routing for Databricks components and integrated services.
ADF Integration Patterns:
Design and implement robust integration patterns between Azure Data Factory (ADF) and Azure Databricks, including notebook activity execution, data movement, and orchestrating complex data pipelines.
Automated Deployments:
Work with Dev Ops teams to define and implement CI/CD pipelines for Databricks assets, notebooks, jobs, and configurations using ADF or Azure Dev Ops.
API Management:
Leverage Databricks REST APIs for automation and integration with other enterprise systems as needed.
Storage Account Design:
Advise on the design and optimal configuration of Azure Data Lake Storage Gen2 accounts for Databricks workloads, including hierarchical name spaces, access tiers, and lifecycle management policies.
Mount Points & Credential Passthrough:
Guide and troubleshoot secure access patterns to ADLS Gen2 from Databricks, including mount points (legacy) and direct access via Entra passthrough or service principals.

7. Operations, Monitoring & Reliability:

Monitoring & Alerting:
Implement comprehensive monitoring for Databricks work spaces using Azure Monitor, Log Analytics, and Databricks native logging, setting up alerts for critical events (e.g., cluster failures, job errors, resource utilization).
Disaster…