Senior Data Platform Engineer

Overview

Senior Data Platform Engineer

Luton/Hybrid

Company

When it comes to innovation and achievement there are few organisations with a better track record. Join us and you’ll be able to play a big part in the success of our highly successful, fast-paced business that opens up Europe so that people can exercise their get-up-and-go. With almost 300 aircraft flying over 1,000 routes to more than 32 countries, we’re the UK’s largest airline, the fourth largest in Europe and the tenth largest in the world.

Set to fly more than 90 million passengers this year, we employ over 10,000 people. Its big-scale stuff and we’re still growing.

Job Purpose

With a big investment into Databricks, and with a large amount of interesting data, this is the chance for you to come and be part of an exciting transformation in the way we store, analyse and use data in a fast paced organisation. You will join as a Senior Platform Data Engineer providing technical leadership to the Data Engineering team. You will work closely with our Data Scientists and business stakeholders to ensure value is delivered through our solutions.

• Develop robust, scalable data pipelines to serve the easy
  
  Jet analyst and data science community.
• Highly competent hands-on experience with relevant Data Engineering technologies, such as Databricks, Spark, Spark API, Python, SQL Server, Scala.
• Work with data scientists, machine learning engineers and Dev Ops engineers to develop, develop and deploy machine learning models and algorithms aimed at addressing specific business challenges and opportunities.
• Coach and mentor the team (including contractors) to improve development standards.
• Work with Business Analysts to deliver against requirements and realise business benefits.
• Build a documentation library and data catalogue for developed code/products.
• Oversight of project deliverables and code quality going into each release.

• Technical Ability: has a high level of current, technical competence in relevant technologies, and be able to independently learn new technologies and techniques as our stack changes.
• Clear communication; can communicate effectively in both written and verbal forms with technical and nontechnical audiences alike.
• Complex problem-solving ability; structured, organised, process-driven and outcome-oriented. Able to use historical experiences to help with future innovations.
• Passionate about data; enjoy being hands-on and learning about new technologies, particularly in the data field.
• Self-directed and independent; able to take general guidance and the overarching data strategy and identify practical steps to take.
• Technical Skills Required
• Significant experience designing and building data solutions on a cloud based, big data distributed system.
• Hands-on software development experience with Python and experience with modern software development and release engineering practices (e.g. TDD, CI/CD), and software deployment automation with Git Hub actions or Azure Dev Ops.
• Experience in testing automation of data transformation pipelines, using frameworks like Pytest or dbt Unit Test.
• Comfortable writing efficient SQL and debugging.
• Data warehouse operations and tuning experience in schema evolution, indexing, partitioning.
• Hands-on IaC development experience with Terraform or Cloud Formation.
• Understanding of ML development workflow and knowledge of when and how to use dedicated hardware.
• Significant experience with Apache Spark or any other distributed data programming frameworks (e.g. Flink, Hadoop, Beam)
• Familiarity with Databricks as a data and AI platform or the Lakehouse Architecture.
• Experience with data quality and/or and data lineage frameworks like Great Expectations, dbt data quality, Open Lineage or Marquez, and data drift detection and alerting.
• Understanding of Data Management principles (security and data privacy) and how they can be applied to Data Engineering processes/solutions (e.g. access management, data privacy, handling of sensitive data (e.g. GDPR)).

• Experience in event-driven architecture, ingesting data in real time in a commercial production environment with Spark…