Lead Data Engineer

Waracle are looking for a Lead Data Engineer for an exciting new role joining our world-class digital technology consultancy and home to a diverse, smart, curious and ambitious community of specialists in technology-driven transformation.

We work with ambitious clients to help them solve their biggest business and customer challenges. We help our clients to innovate and create intelligent digital products and services. We thrive on complex challenges and deliver business-critical IT transformation projects, moving seamlessly from strategy, design and delivery to operations.

This is a Hybrid role working from any of our offices (Glasgow, London, Edinburgh or Dundee) 2 days a week

As a Lead Data Engineer, you will be a critical technical leader, accountable for the technical delivery and overall quality of large-scale data and AI projects. You will act as a consultant, setting the technical direction to meet complex client requirements and translating business priorities into actionable technical strategy. This role requires a strong balance of technical expertise (ensuring performance, cost, and security are balanced in system designs) and leadership (coaching, conflict de-escalation, and driving improvements across multiple teams).

You will ensure technical decisions consistently align with client business objectives, making proactive tradeoffs on scope or resources to maintain delivery pace.

• Owning and directing technical delivery for large-scale data projects, maintaining accountability for technical outcomes, solution quality, and proactively making tradeoffs on scope or resources to maintain delivery pace.

• Defining technical strategy by translating client priorities into actionable plans, and ensuring team delivery consistently aligns with business objectives.

• Designing robust data systems while effectively balancing performance, cost, and security requirements.

• Driving best practices by shaping CI/CD and IaC practices across multiple teams, and defining coding, testing, and review standards across the Data & AI practice.

• Resolving complex challenges including sophisticated integration issues across cloud and on-prem systems, and leading root cause analysis and systemic fixes for major incidents.

• Partnering and aligning stakeholders by working with Product/Delivery to align technical decisions with client outcomes, managing expectations on scope and budget, and de-escalating conflicts.

• Leading project execution by facilitating backlog refinement and workshops, guiding the team in tracking delivery metrics (e.g., DORA), and ensuring clear, up-to-date technical documentation is created.

• Leading and coaching the team by ensuring a collaborative, psychologically safe environment, coaching team members regularly, and formally/informally mentoring others in the practice.

• Upholding Waracle’s values on client engagements and driving cross-project improvements in delivery tooling.

• Proven track record of designing, implementing, and optimising complex, large-scale data pipelines and cloud infrastructure

• Deep expertise in at least one major cloud platform (AWS, Azure, or GCP), with working knowledge of a second platform preferred

• Advanced programming skills in Python and SQL, with experience in Scala or Java

• Expert-level knowledge of Infrastructure as Code tooling (Terraform, Cloud Formation, ARM templates)

• Comprehensive experience with data processing frameworks (Apache Spark, dbt) and orchestration tools (Apache Airflow, Step Functions)

• Strong command of CI/CD tools and methodologies (Git Hub Actions, Azure Pipelines, Cloud Build)

• Demonstrable expertise in system design, technical planning, and architectural guidance for distributed data systems

• Exceptional client-facing and consulting abilities, with strong presentation, stakeholder management, and mentoring skills

• Strong understanding and practical application of Agile methodologies (Scrum, Kanban)

The recruitment process you can expect for this role is an initial call with your dedicated Talent Acquisition Partner who will chat with you about Waracle, what you are looking for in a new position, the salary for the…