Lead Architect - Azure DBX

Senior Azure Databricks Data Engineer

Fractal is a strategic AI partner to Fortune 500 companies, focused on building enterprise‑scale data platforms on Microsoft Azure.

We are seeking a highly experienced senior engineer to design, develop and optimise large‑scale data solutions on Azure Databricks. The ideal candidate will lead the migration and modernization of legacy data platforms to Azure, architect Lakehouse solutions, and provide technical mentorship across teams.

• Design and implement scalable, high‑performance data platforms using Azure Databricks and other Azure services.
• Develop robust ETL/ELT pipelines with PySpark, Spark SQL, and Databricks workflows.
• Architect and implement Lakehouse solutions leveraging Delta Lake and the Medallion architecture.
• Lead complex data migration and modernization projects from on‑premises and legacy platforms to Azure.
• Build and maintain batch and real‑time data processing solutions.
• Integrate data pipelines with Azure Data Factory, ADLS Gen2, Azure Synapse Analytics, Azure SQL Database and other Azure services.
• Optimize Spark jobs, cluster utilisation, partitioning strategies and query performance.
• Establish data quality, monitoring, governance and security frameworks.
• Implement CI/CD pipelines using Azure Dev Ops and Git‑based source control.
• Mentor junior engineers and provide technical leadership across project teams.
• Collaborate with architects, stakeholders and business teams to define data strategies and technical roadmaps.
• Ensure adherence to enterprise security, compliance and governance standards.

• 10+ years of experience in Data Engineering, BI or Data Warehousing.
• 5+ years of hands‑on experience with Azure Databricks.
• Strong expertise in PySpark, Spark SQL, Python and advanced SQL.
• Experience designing enterprise‑scale Data Lakes and Lakehouse architectures.
• Hands‑on experience with Delta Lake, Delta Live Tables (DLT) and Unity Catalog.
• Extensive experience with Azure Data Factory (ADF).
• Strong knowledge of Azure Data Lake Storage (ADLS Gen2).
• Experience with Azure Synapse Analytics and Azure SQL Database.
• Expertise in performance tuning, troubleshooting and Spark optimisation.
• Strong understanding of data modelling, data warehousing concepts and dimensional modelling.
• Azure Dev Ops CI/CD implementation experience.
• Git version control and branching strategies.
• Knowledge of Infrastructure as Code (Terraform, ARM, or Bicep).
• Knowledge of environment migration and deployment automation.
• Data governance and metadata management experience.
• RBAC, Unity Catalog, credential passthrough and security best practices.
• Knowledge of compliance frameworks and enterprise data security standards.

• Databricks Workflows and Delta Live Tables.
• Streaming frameworks using Structured Streaming, Event Hub or Kafka.
• PowerBI integration.
• Data Mesh and modern data architecture principles.
• Experience with machine‑learning workflows within Databricks.
• Experience working in Agile/Scrum environments.

• Bachelor’s or Master’s degree in Computer Science, Information Technology, Engineering or related discipline.
• Certifications (preferred):
  • Databricks Certified Data Engineer Professional
  • Databricks Certified Data Engineer Associate
  • Microsoft Certified:
    Azure Data Engineer Associate (DP‑203)
  • Microsoft Azure Solutions Architect Expert

• Technical leadership and solution architecture capabilities.
• Strong stakeholder management and communication skills.
• Problem‑solving and decision‑making abilities.
• Ability to drive technical discussions and mentor teams.
• Ownership mindset and commitment to delivery excellence.

Senior Engineer / Lead Data Engineer capable of owning end‑to‑end Azure Databricks solutions, guiding teams, and driving enterprise data transformation initiatives.

The compensation range for this role is $180,000–$190,000, with potential discretionary bonuses. Compensation is adjusted on a case‑by‑case basis.

• Health, dental, vision, life insurance and disability plans.
• Eligibility for benefits on the first day of employment.
• 401(k) Plan after 30days of employment.
• 11 paid holidays and 12weeks of parental leave.
• Paid time off (PTO) policy for sick time or vacation.

Fractal provides equal employment opportunities to all employees and applicants for employment and prohibits discrimination and harassment of any type without regard to race, color, religion, age, sex, national origin, disability status, genetics, protected veteran status, sexual orientation, gender identity or expression, or any other characteristic protected by federal, state or local laws.