Principal Digital Specialist - Principal Data Engineer; Azure

Together we are Trusted to Serve Scotland.

Principal Data Engineer (Azure)

We are seeking a Principal Data Engineer to join our Business Analytics team and lead the development of our cloud-based data platform. Our team’s mission is to turn data into actionable insights that support better decision‑making and deliver measurable value for the business. To enable this, we have built an Azure data platform built on lakehouse architecture. The platform leverages Fabric and Azure Databricks and provides a secure, governed and trusted foundation for our data.

The platform offers a consistent and safe environment to develop, deploy and scale data products and AI/ML solutions, ensuring insights can be shared efficiently and used with confidence across the organisation.

We are an ambitious team that uses data, advanced analytics and cloud technologies to design and deliver digital solutions that help colleagues drive improvements in key business areas, including asset management and water quality. We work closely with stakeholders through an inclusive and collaborative approach, ensuring we focus on the right problems and deliver outcomes that are genuinely useful. Our Business Analytics function brings together a strong pool of talent and provides excellent opportunities for learning, development and growth.

The role

As Principal Data Engineer, you will be the senior technical authority for data engineering in our organisation. This is a hands-on leadership role with no direct line‑management duties, where you will set the standards and technical direction for our data engineering practice. You will guide the design and implementation of robust data pipelines and architectures on Azure, ensure high‑quality engineering outcomes through code reviews and mentoring, and spearhead innovation in tools and techniques.

In this role, you will work closely with data scientists, solution developers, enterprise architects and platform engineers to ensure our data ecosystem is reliable, scalable and delivers value. You will also act as the key technical liaison with our external Digital partners, ensuring alignment with our standards and providing technical assurance on joint initiatives. This is a fantastic opportunity to shape the future of data engineering in a forward‑thinking, data‑driven organisation while staying at the forefront of industry developments.

What you will do

You will own and evolve data engineering standards and guiding technical principles across the Azure data analytics estate, acting as the technical authority for data engineering and ensuring high‑quality, well‑assured engineering outcomes. This includes gatekeeping pull requests in Azure Dev Ops for changes to production data pipelines and leading the research and development of new tools, patterns and platform capabilities within the analytics environment.

You will design and assure technical designs for all projects on the Analytics platform and undertake hands‑on development for proof‑of‑concept work and complex technical initiatives. Collaboration is central to the role, and you will work closely with analytics, data science, architecture and platform teams, as well as with the Platform Product Owner and Technical Support Team Lead to transition established solutions into support.

What you'll bring

Essential skills and experience

You will bring significant hands‑on experience as a senior or principal Data Engineer on Microsoft Azure, along with strong experience of modern Azure data platforms such as Azure Data Lake, Microsoft Fabric and Databricks. You will have a proven ability to design and build scalable, secure data pipelines and lakehouse architectures, supported by strong programming skills, for example Python and SQL, with Spark experience desirable.

You will have expertise working within Dev Ops CI/CD environments using Azure Dev Ops and a solid understanding of data quality, security and governance in regulated environments. Excellent communication skills are essential, along with the ability to influence through technical expertise.

Desirable

Experience with streaming or IoT data would be advantageous, along with familiarity with…