Data Engineer

This is an exciting role that will allow you to be integral to the evolution of our cloud-based enterprise data platform. As a Data Engineer, you will be working on technical solutions, whilst supporting our existing set of SQL Servers that are predominantly used to feed data into a number of analytics products. This will help the business with data-led decision making to provide essential public services.

• 1. Designing efficient and robust data engineering projects within both on-premises and cloud-based data stacks that add business value and comply with architectural and data security requirements.
• 2. Building, maintaining and supporting data engineering pipelines, with resolution of incidents.
• 3. End-to-end delivery of data engineering solutions, including data quality, testing, peer reviewing, documenting and demonstrating of completed solutions to the team.
• 4. Working closely with teams within data and other technical colleagues across the business.
• 5. Remaining aware of new data engineering approaches, and be able to suggest how the latest research, techniques and approaches could be implemented to achieve business benefit
• 6. Undertaking any other duties as required to meet the needs of the business.

• 1. Substantial experience of data pipeline design and management
• 2. Enthusiasm to add customer value whilst improving the effectiveness of the data engineering function
• 3. Proactive problem solving and being able to effectively communicate solutions to both the data team, and to stakeholders
• 4. Good understanding of database administration activities and principles
• 5. Data transformation approaches
• 6. Strong Python and SQL programming capabilities
• 7. Data security approaches - e.g. permission models (AD, role-based etc.) and techniques to protect data in certain circumstances (e.g. encryption, masking)

• 1. Familiarity with common agile project management principles, particularly in the context of data and analytical project delivery
• 1. Good understanding of typical ingestion patterns (e.g. ETL, ELT) and their effective implementation with on-premises and cloud-based environments
• 2. Knowledge of typical data modelling approaches (e.g. Kimball)
• 3. Experience using the Microsoft data stack (especially Azure Data Lake, Azure Synapse, Azure Data Factory, Databricks)
• 4. Experience of the full software/reporting lifecycle from planning and design through to deployment and maintenance
• 5. Knowledge of the Microsoft BI Stack, including Analysis Services, Integration Services and Reporting Services, and Power BI
• 6. Understanding of Dev Ops (or other source control technologies) - particularly code repositories and version control, containerisation methodologies, and application deployment practices (CI/CD pipelines)