Solution Architect - Databricks

Position: Solution Architect - Databricks )
As a  Solution Architect – Databricks , you will design and deliver enterprise‑grade data solutions on Azure Databricks, leading a large‑scale data modernisation programme for a telecommunications client. You will architect migration strategies, define governance frameworks, and provide technical leadership to ensure seamless transformation from legacy data warehousing systems to a modern lakehouse architecture.

What You’ll Do and How You’ll Succeed

Architecture & Design
Design and architect end‑to‑end data pipelines using Azure Databricks with Medallion Architecture (Bronze Silver Gold Semantic layers).
Lead hybrid orchestration frameworks combining DLT‑Lakeflow for CDC ingestion and metadata‑driven frameworks for transformation layers.
Define and implement Unity Catalog governance strategies for data access, security, and lineage.
Architect Databricks Workflows for job scheduling, dependency management, and replacing legacy Control‑M orchestration.
Design SCD Type 2 implementations and complex transformation patterns for EDW migration.

Technical Leadership
Provide guidance on PySpark and Spark SQL best practices for large‑scale data processing.
Define coding standards, modular notebook organisation, and configuration‑driven development approaches.
Lead technical decisions on schema evolution, data quality frameworks, and reconciliation strategies.
Recommend approaches for surrogate key generation, batch control mechanisms, and selective reprocessing.

Migration & Modernisation
Architect migration strategies from legacy systems (ODS, EDW, Informatica Power Center) to Azure Databricks lakehouse.
Design CDC ingestion patterns using Oracle Golden Gate and Informatica IDR integration.
Define approaches for historical data migration and parallel run strategies.
Ensure minimal refactoring of existing SQL/PySpark code through metadata‑driven frameworks.

Operational Excellence
Design solutions supporting BAU operations including manual data patching, hold/resume/skip controls, and entity‑level reprocessing.
Architect job/cycle/batch auditing frameworks aligned with existing operational patterns.
Define monitoring, alerting, and logging strategies across all pipeline layers.
Ensure solutions support targeted fixes without requiring full refreshes.

Stakeholder Engagement
Collaborate with Amdocs and customer architecture teams on design approvals and technical alignment.
Participate in discovery sessions for EDW entities (DDS, SDS, DNF).
Present technical solutions and trade‑off analyses to leadership and steering committees.
Work closely with delivery teams to translate designs into implementable solutions.

We’d Love to Hear From You If…

Experience
You have 8+ years in data engineering and architecture roles.
You have 4+ years of hands‑on experience with Databricks (Delta Lake, Spark, Unity Catalog).
You have 3+ years of experience with Azure cloud services (ADLS Gen2, Azure Dev Ops, Azure Data Factory).
You have proven experience in large‑scale data migration projects, ideally EDW modernisation.

Technical Expertise
You are expert in Azure Databricks (Delta Lake, DLT/Lakeflow, Databricks Workflows, Unity Catalog).
You are proficient in PySpark and Spark SQL for optimised transformations.
You have hands‑on experience with Medallion Architecture (Bronze/Silver/Gold).
You are skilled in dimensional modelling, SCD implementations, and surrogate key strategies.
You are experienced in metadata‑driven ETL/ELT frameworks.
You have worked with CDC tools such as Oracle Golden Gate or Kafka.
You are familiar with data quality frameworks, reconciliation, and exception handling.
You are experienced with Git, CI/CD pipelines, and Databricks Asset Bundles (DAB).
Preferred:
Power BI, Informatica Power Center/IDR, Control‑M, Terraform/IaC, telecommunications domain.

Ways of Working
You have strong analytical and problem‑solving abilities.
You communicate complex technical concepts clearly to diverse audiences.
You collaborate effectively across distributed teams and time zones.
You mentor and guide junior team members.
You document design decisions thoroughly (HLSD, LLD).
You bring a collaborative mindset when working with customer stakeholders.

Certifications…