Data Pipeline Engineer Job Glasgow Scotland UK,Software Development

Data Pipeline Engineer / Data Pipeline Architect

* 3-month contract (initially)

* Remote Working (UK-based)

* Market Rates

* Outside IR35

The Opportunity

We are supporting a central government client who are seeking an experienced Data Pipeline Engineer and Architect to help design and build an open-source Data Transformation Layer for a national data exchange platform.

This is a hands-on engineering and architecture role, combining the design of reusable, reliable data pipelines with practical development using Apache NiFi, RDF, SPARQL, Python and Java.

The platform will use open-source, cloud-agnostic technologies to ingest, transform, validate and exchange data securely between systems.

Skills and Experience

Proven experience designing and delivering data integration or ETL solutions using Apache NiFi.
Practical project experience with RDF triples, SPARQL and semantic data modelling.
Hands-on development experience in both Python and Java.
Experience designing reusable pipelines across APIs, databases, files and other data sources.
Understanding of reliable data processing, validation, automated testing and operational monitoring.
Experience with version control, CI/CD, secure integrations and repeatable deployments.
Ability to translate high-level requirements into technical designs and production-ready solutions.

Desirable Experience

Apache Jena and Jena Fuseki.
Apache Kafka and event-driven data integration.
SHACL or other RDF validation approaches.
Custom NiFi processor development.
Docker, Kubernetes and portable deployment architectures.
Large RDF datasets, SPARQL optimisation or secure data exchange between organisations.

Key Technologies

Role and Responsibilities

Design and evolve the architecture of a reusable, cloud-agnostic data transformation platform.
Build, test and maintain Apache NiFi flows, including integrations with APIs, databases, files and messaging systems.
Develop pipelines to ingest, transform, validate and publish RDF triples, using SPARQL for querying and updates.
Develop supporting components and integrations using Python and Java.
Define data mappings, interface contracts, technical specifications and architectural decisions.
Implement robust processing, including error handling, retries, duplicate management and recovery.
Establish automated testing, monitoring, version control and repeatable deployment practices.
Collaborate with architects, developers and stakeholders to translate requirements into working solutions.
Produce clear technical documentation and support the delivery of reusable platform capabilities.

NB:
Candidates must be eligible for BPSS security clearance which will processed following successful interviews (5-7 days).