Databricks Architecture

Solution Architect Databricks Lakehouse Migration AI Modernization Raritan NJ Onsite We are seeking an experienced Solution Architect to lead the migration and modernization of enterprise scale data lakes from Azure Databricks to AWS Databricks while transforming the platform into an AIready Lakehouse architecture The role involves defining target state architecture migration strategy governance frameworks modernization roadmaps and AIGenAI enablement across large scale data ecosystems

The ideal candidate will have deep expertise in Databricks cloud migration modern data architecture and AIdriven data platforms with experience leading complex enterprise transformation programs involving thousands of data assets and multiple downstream applications

Key Responsibilities Architecture Strategy Define endtoend migration strategy from Azure Databricks to AWS Databricks Design scalable Lakehouse architecture using Medallion principles Bronze Silver Gold Develop Day1 Business Continuity and Day2 Modernization AI Enablement migration roadmaps

Perform application and dependency assessment across upstream and downstream systems

Establish architecture patterns standards and best practices for cloud native data platforms

Migration Modernization Lead migration of data pipelines notebooks Delta tables workflows and data products

Modernize legacy ETLELT frameworks using Databricks native capabilities

Optimize performance scalability security and operational efficiency

Define migration approach for CDC batch streaming and realtime workloads

Drive technology rationalization and tool consolidation initiativesAIReady Data Platform Build AIready data foundations leveraging Delta Lake and Unity Catalog Design data architectures supporting ML GenAI RAG feature stores and advanced analytics

Enable governed data access for data scientists and AI applications

Define semantic models metadata strategies lineage and data discovery frameworks

Establish frameworks for AI governance and responsible AI adoption

Data Governance Security Architect enterprise wide governance using Unity Catalog Design data lineage auditing access controls masking and compliance standards

Implement data quality observability and monitoring frameworks

Ensure platform adherence to security and regulatory requirements

Stakeholder Management Collaborate with business data engineering analytics AI security and cloud teams

Lead architecture review boards and executive level discussions

Mentor solution architects technical leads and engineering teams

Drive estimation roadmap planning risk assessment and governance reviews

Required Experience
12 years in Data Analytics5 years as a Solution Architect or Lead Architect Proven experience in large scale cloud migration programs

Handson experience with Azure Databricks and AWS Databricks Experience migrating enterprise Data Lakes Lake houses across cloud platforms

Strong understanding of Medallion Architecture and Delta Lake

Experience with Modern Data Platforms supporting AIML workloads

Expertise in enterprise integration and large scale data ecosystems

Technical Skills Databricks Data Engineering Databricks Workflows Delta Lake Delta Sharing Unity Catalog Delta Live Tables Spark Py Spark Databricks  SQL Structured Streaming Cloud Platforms Azure Databricks AWS DatabricksADLS Gen2

Amazon S3

IAMVPCGlue Lambda Redshift  good to have

Data Integration Informatica Power Center IDMCAzure Data Factory

KafkaCDC Frameworks API Integration Event driven Architectures AI Advanced Analytics

GenAI Architecture RAG Frameworks Vector Databases Feature StoresMLflow

Databricks AIMLAzure OpenAI Amazon Bedrock Dev Ops  GovernanceCICDGit Terraform Infrastructure  as Code Data Quality Frameworks Data Observability Security Compliance

Preferred Qualifications Data…