Senior Data Engineer; Temp

Pantheon has been at the forefront of private markets investing for more than 40 years, earning a reputation for an innovative approach to investing in secondaries, co‑investments, and primary fund investments, as well as capital formation across commingled funds, evergreen vehicles and customized solutions. Our specialist investment capabilities span multiple strategies across private equity, infrastructure and real assets, and private credit.

Through our collaborative and committed culture, we find new ways to solve complex problems together and deliver innovative investment opportunities across private markets. Pantheon currently manages approximately $82.3 billion in AUM across all its strategies, serving more than 750 institutional and 638 private wealth clients worldwide.

Pantheon is in the process of building a cloud‑native, AI‑ready Data Platform based on the Databricks Lakehouse architecture, enabling analytics, operational use cases, and advanced ML/AI workloads. We require an experienced and passionate hands‑on Senior Data Engineer to design and implement new data pipelines for adaptation to business and/or technology changes. This role will be integral to the success of this program and establishing Pantheon as a data‑centric organisation.

You will be working with a modern Azure tech stack and proven experience of ingesting and transforming data from a variety of internal and external systems is core to the role.

You will be part of a small and highly skilled team, and you will need to be passionate about providing best in class solutions to our global user base.

• Design, build, and maintain scalable, secure, and high‑performance data pipelines on Azure, primarily using Azure Databricks, Azure Data Factory, and Azure Functions.
• Develop and optimise batch and streaming data processing solutions using PySpark and SQL to support analytics, reporting, and downstream data products.
• Implement robust data transformation layers using dbt, ensuring well‑structured, tested, and documented analytical models.
• Collaborate closely with business analysts, QA teams, and business stakeholders to translate data requirements into reliable technical solutions.
• Ensure data quality, reliability, and observability through automated testing, monitoring, logging, and alerting.
• Lead on performance tuning, cost optimisation, and capacity planning across Databricks and associated Azure services.
• Implement and maintain CI/CD pipelines using Azure Dev Ops, promoting best practices for version control, automated testing, and deployment.
• Enforce data governance, security, and compliance standards, including access controls, data lineage, and auditability.
• Contribute to architectural decisions and provide technical leadership, mentoring junior engineers and setting engineering standards.
• Produce clear technical documentation and contribute to knowledge sharing across the data engineering function.

• Python and PySpark for large-scale data processing.
• SQL (advanced querying, optimisation, and data modelling).
• Azure Data Factory (pipeline orchestration and integration).
• Azure Dev Ops (Git, CI/CD pipelines, release management).
• Lakehouse architecture (Databricks Unity Catalog, Delta Lake optimisation techniques such as Z‑ordering, liquid clustering).
• Data modelling (star schemas, data vault, or lakehouse‑aligned approaches).
• Data quality, testing frameworks, and monitoring/observability.
• Strong problem‑solving ability and a pragmatic, engineering‑led mindset.
• Experience in Agile SW development environment.
• Excellent communication skills, with the ability to explain complex technical concepts to both technical and non‑technical stakeholders.
• Leadership and mentoring capability, with a focus on raising engineering standards and best practices.
• Significant commercial experience (typically 5+ years) in data engineering roles, with demonstrable experience designing and operating production‑grade data platforms.
• Strong hands‑on experience with Azure Databricks, including cluster configuration, job orchestration, and performance optimisation.
• Proven experience building…