Lead Architect - Azure DBX

Senior Azure Databricks Data Engineer

Fractal is a strategic AI partner to Fortune 500 companies with a vision to power every human decision in the enterprise. Fractal is building a world where individual choices, freedom, and diversity are the greatest assets; an ecosystem where human imagination is at the heart of every decision. Where no possibility is written off, only challenged to get better. We believe that a true Fractalite is the one who empowers imagination with intelligence.

Fractal has been featured as a Great Place to Work by The Economic Times in partnership with the Great Place to Work® Institute and recognized as a 'Cool Vendor' and a 'Vendor to Watch' by Gartner.

Role:
Senior Azure Databricks Data Engineer

Role Summary

We are looking for a highly experienced Senior Azure Databricks Data Engineer with 10+ years of IT experience to lead the design, development, and optimization of enterprise-scale data platforms on Microsoft Azure. The ideal candidate will have deep expertise in Azure Databricks, PySpark, Delta Lake, cloud-native data architectures, and large-scale data modernization initiatives.

Key Responsibilities

• Design and implement scalable, high-performance data platforms using Azure Databricks and Azure cloud services.
• Develop robust ETL/ELT pipelines using PySpark, Spark SQL, and Databricks workflows.
• Architect and implement Lakehouse solutions leveraging Delta Lake and Medallion architecture.
• Lead complex data migration and modernization projects from on-premises and legacy platforms to Azure.
• Build and maintain batch and real-time data processing solutions.
• Integrate data pipelines with Azure Data Factory (ADF), ADLS Gen2, Azure Synapse Analytics, Azure SQL Database, and other Azure services.
• Optimize Spark jobs, cluster utilization, partitioning strategies, and query performance.
• Establish data quality, monitoring, governance, and security frameworks.
• Implement CI/CD pipelines using Azure Dev Ops and Git-based source control.
• Mentor junior engineers and provide technical leadership across project teams.
• Collaborate with architects, stakeholders, and business teams to define data strategies and technical roadmaps.
• Ensure adherence to enterprise security, compliance, and governance standards.

Required Technical Skills

• 10+ years of experience in Data Engineering, BI, or Data Warehousing.
• 5+ years of hands-on experience with Azure Databricks.
• Strong expertise in PySpark, Spark SQL, Python, and advanced SQL.
• Experience designing enterprise-scale Data Lakes and Lakehouse architectures.
• Hands-on experience with Delta Lake, Delta Live Tables (DLT), and Unity Catalog.
• Extensive experience with Azure Data Factory (ADF).
• Strong knowledge of Azure Data Lake Storage (ADLS Gen2).
• Experience with Azure Synapse Analytics and Azure SQL Database.
• Expertise in performance tuning, troubleshooting, and Spark optimization.
• Strong understanding of data modeling, data warehousing concepts, and dimensional modeling.

Cloud & Dev Ops

• Azure Dev Ops CI/CD implementation experience.
• Git version control and branching strategies.
• Knowledge of Infrastructure as Code (Terraform, ARM, or Bicep).
• Experience with environment migration and deployment automation.

Governance & Security

• Data governance and metadata management.
• RBAC, Unity Catalog, credential passthrough, and security best practices.
• Knowledge of compliance frameworks and enterprise data security standards.

Preferred Skills

• Databricks Workflows and Delta Live Tables.
• Streaming frameworks using Structured Streaming/Event Hub/Kafka.
• Experience with Power BI integration.
• Knowledge of Data Mesh and modern data architecture principles.
• Exposure to Machine Learning workflows within Databricks.
• Experience working in Agile/Scrum environments.

Education

• Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or related discipline.

Certifications (Preferred)

• Databricks Certified Data Engineer Professional
• Databricks Certified Data Engineer Associate
• Microsoft Certified:
  Azure Data Engineer Associate (DP-203)
• Microsoft Azure Solutions Architect Expert

Key Competencies

• Technical leadership and solution architecture capabilities.
• Strong stakeholder management and…