Data Engineer

Select how often (in days) to receive an alert:

Riverside Tower, Belfast (N.I), United Kingdom

All Locations:
London, Belfast

Working Style: 3 days a week in office, 2 days from anywhere

Data is a vital part of our business and with your help, we can improve the services that we provide to our customers through engineering excellence of our data products. You’ll be working on exciting data products which support TV, Broadband, Digital Voice and market leading technologies like Unbreakable Wi-Fi. We deliver these critical services using the latest cloud technologies on AWS, GCP and Cloudera.

When you join us as a lead data engineer, you’ll have the opportunity to be part of our Data & AI Enablement Team where you will help to design, build, and support high quality engineering products from our extensive data sources and multi-cloud environment.

As a team we believe in the agile principles of openness, transparency and continuous improvement, which underpin strong relationships with our business stakeholders.

There will be lots of opportunities to explore new technologies, develop new skills, innovate and grow as an engineer.

• Design, develop, and maintain end-to-end data pipelines using Ab Initio (GDE, PDL, Conduct>
  
  IT, Express>
  
  IT).
• Build scalable batch and streaming workloads using Apache Spark (PySpark/Scala).
• Develop and orchestrate data ingestion flows using Apache Ni Fi for real-time and near-real-time use cases.
• Optimize ETL/ELT jobs for scalability, throughput, and resource efficiency.
• Perform bottleneck analysis and tuning for Spark jobs, NiFi processors, and Ab Initio graphs.
• Demos and knowledge transfers to stakeholders around product developments.
• Spread your experience and knowledge to junior engineers.
• Improve automated testing and deployments using CI/CD pipelines.
• Expectation to line manage small team of junior data engineers.
• Continual learning through internal and external training.

• Proficient in building Big Data solutions using Hadoop or GCP or AWS.
• Skilled in technologies such as Scala, Java, Python, Spark (PySpark/Scala), Kafka, Kinesis, Big Query, Dataflow, Big Table, Terraform, and SQL.
• Experience with Terraform, Kafka, Kinesis, SQL are nice to have
• Strong expertise in Ab Initio (GDE, PDL, Conduct>
  
  IT, Express>
  
  IT, EME) with a focus on parallel processing and component optimisation.
• Hands-on experience with Apache Spark, including Spark SQL, Spark Streaming, and performance tuning.
• Practical experience with Apache NiFi for flow design, processor tuning, error handling, and security configurations.
• Good understanding of Infrastructure as Code toolsets.
• Experienced in implementing Continuous Integration and Deployment strategies.
• Proficient in using Atlassian tools such as Confluence and JIRA.
• Experienced in version control using Git or SVN.
• Able to clearly present technical issues, progress, and outcomes to team members and Product Owners.
• Demonstrates enthusiasm for learning and applying new technologies.
• Experience or interest in managing and developing team members.
• Capable of building new solutions as well as supporting existing data products and platforms.

• 10% on target bonus
• Life Assurance Cover
• Exclusive colleague discounts on our latest and greatest BT broadband packages, BT TV with TNT Sports and NOW Entertainment
• Equal family leave: receive 18 weeks at full pay, 8 weeks at half pay and 26 weeks at the statutory rate. It’s for all parents, no matter how your family is made up.
• Enhanced women’s health support: including help with menopause symptoms, cancer screenings, period care and more.
• 25 days annual leave (not including bank holidays), increasing with service
• 24/7 private virtual GP appointments for UK colleagues
• 2 weeks carer’s leave
• World-class training and development opportunities
• Option to join BT Shares Saving schemes
• EE Broadband – from £10/month for 150
  
  Mb and £20 for 900
  
  Mb – lower than any other provider.
• EE Mobile – Save 50% on Mobile & SIM-only plans or pay just £10/month for an EE SIM.
• EE TV – Enjoy premium content like Netflix and TNT Sports, with the flexibility to swap packages monthly, starting from just…