Data Platform Tech Lead

Data Platform Tech Lead

Department: Engineering

Employment Type: Permanent - Full Time

Location: Edinburgh

Reporting To: Dave Evans

Compensation: £59,000 - £87,000 / year

Our Data Platform Team The mission of the Data Platform team is to empower Free Agent’s engineers to reliably, securely, and sustainably derive value from our data assets.

The role We’re looking for a Tech Lead who will play an active part in the design and implementation of our data platform. You will lead a small dedicated team of engineers, ensuring they're well directed, motivated, productive and engaged in their work. You will collaborate closely with the Data Science and Analytics Engineering teams to ensure they have the right tools for their jobs, and enable stakeholders across Free Agent to turn messy data into valuable insights.

This is a team leadership role with a focus on technical contribution.

Our past projects include the design and delivery of an event system which processes over 100 million events per month, automating metadata management in our data lake, and implementing mechanisms for running regular quality checks on our data. Our platform supports our machine learning based automations for document extraction, bill, expense and bank transaction categorisation. We categorise over two million bank transactions every month with accounting grade accuracy.

Currently, we’re in the process of migrating to a new orchestration tool, which presents exciting opportunities to shape Free Agent’s data tech stack and make a positive impact by bringing modern engineering practices to our data pipelines.

Day to day your role will involve

• Helping us realise our vision for our next generation data platform to make it easy to collect, process, store and analyse customer and business data
• Leading an agile development process for your team
• Collaborating cross-functionally with other engineering teams
• Assisting in producing common development and architectural standards and ensure they're upheld across engineering teams
• Cutting code alongside your team when required, review pull requests
• Line managing your team, and onboard, coach and mentor any new hires
• Keeping up to date with developments in data, ML and AI technology and techniques
• Working with senior engineering leadership to understand the big picture and the future vision of the Free Agent stack and drive forward the development of the data platform to meet these goals

Our data platform is built in AWS and some of the specific tools we use are outlined below. However if you have experience with equivalent tools you should still apply. Recruitment is a great chance for us to bring alternate perspectives into the team!

Ideally you'll have the following skills and experience:

Data Engineering

• You can demonstrate hands‑on experience with a modern data pipeline stack. We’re currently migrating to Dagster, dbt and dlt and we work with structured and unstructured data from a variety of sources, as well as internal event data
• You have designed and implemented data lakehouse storage architectures using tools like S3, Redshift, Glue, and Athena together with your team
• You can employ an appropriate data modelling technique for a given problem. As well as working with semi‑structured event data we rely on dimensional models for business reporting.
• You have up‑to‑date knowledge of data management and retention regulations, security and ethics

Software development

• You are comfortable following a modern software development methodology to implement and test changes to complex systems. We use Python, Ruby and Git/Github.

Cloud infrastructure

• Experience building and managing infrastructure as code (we use Terraform)
• Experience deploying containerised services with Docker and Git Hub Actions
• Experience with machine learning platforms such as Sage Maker

Leadership and management

• A comprehensive understanding of lean and agile development methodologies
• Experience of or enthusiasm to develop people management skills
• Keeping up to date with new techniques, technologies and resources

Please also make sure you're familiar with the hiring process for this role so you can ensure your application contains all…