Senior Data Platform Engineer

We are seeking an experienced and forward‑thinking Senior Data Platform Engineer to manage, optimize, and evolve the cloud‑native platforms used by the Data Office. The role combines deep data engineering capability with platform engineering accountability across Microsoft Azure, Databricks, Unity Catalog, integration and ingestion services, Collibra, and Immuta. The successful candidate will ensure that the Data Office platform estate is secure, reliable, scalable, observable, and cost‑effective.

This includes optimizing performance, workloads, storage, compute, access, and platform configurations, while also designing and delivering robust data pipelines and lakehouse solutions that support analytics, business intelligence, AI, and machine learning. This is a hands‑on role working across engineering, architecture, governance, security, and operations to provide a trusted enterprise data platform and a strong developer and consumer experience.

The Senior Data Platform Engineer will be responsible for the day‑to‑day engineering and technical management of the Data Office platform ecosystem. The role will maintain platform health, availability, security, performance, and cost efficiency, while supporting platform upgrades, configuration, automation, monitoring, access management, and incident resolution. In parallel, the role will deliver senior‑level data engineering outcomes across ingestion, integration, transformation, orchestration, governance, and data access.

The engineer will work with data architects, engineers, analysts, governance teams, security teams, and business stakeholders to ensure that data products are reliable, discoverable, controlled, and fit for purpose.

• Manage the Data Office platform estate across Azure, Databricks, Unity Catalog, integration and ingestion services, Collibra, and Immuta, ensuring operational reliability and alignment with enterprise standards.
• Monitor and optimize platform performance, workloads, clusters, SQL warehouses, jobs, storage, and compute to improve reliability, scalability, user experience, and cost efficiency.
• Own capacity and workload management, including resource allocation, concurrency, job scheduling, autoscaling, workload isolation, and the assessment of serverless and cluster‑based compute options.
• Administer platform configuration, work spaces, environments, service connections, secrets, policies, and technical integrations across development, test, and production.
• Manage data access controls and entitlements across Azure, Databricks, Unity Catalog, Collibra, and Immuta, applying least‑privilege, role‑based, attribute‑based, and policy‑based access principles.
• Implement and maintain Unity Catalog structures, including catalogs, schemas, permissions, data objects, lineage, and controlled data‑sharing patterns.
• Support Collibra integration and adoption for cataloging, metadata, business glossaries, lineage, ownership, stewardship, and governance workflows.
• Design, build, and maintain secure, scalable lakehouse solutions using Azure Data Lake Storage, Databricks, Delta Lake, Spark, PySpark, Python, and SQL.
• Develop and optimize batch and streaming ingestion, integration, ETL, and ELT pipelines using Azure Data Factory, Databricks Workflows, APIs, event‑based services, and appropriate orchestration tools.
• Engineer reusable integration and ingestion patterns for structured, semi‑structured, and unstructured data from enterprise applications, files, databases, APIs, and event streams.
• Implement observability across platforms and pipelines, including monitoring, logging, alerting, performance metrics, usage reporting, cost reporting, failure analysis,…