Senior Software Engineer - ETL Pipeline Orchestration Platform

Job Overview

Bloomberg is a global leader in business and financial information, delivering trusted data, news, and insights that bring transparency, efficiency, and fairness to capital markets. The Data Technology Infrastructure Department designs and builds large‑scale platforms to acquire financial reference and pricing data globally from various sources, delivering billions of data points daily to clients for investment analysis.

Location:

London
Business Area:
Engineering and CTO

The ETL Pipeline Orchestration Platform team owns a low‑code platform that helps engineers and analysts author and manage data transformation, derivation, validation and database loading themselves. As adoption accelerates, we are redesigning key components to support further throughput and resiliency while maintaining system availability.

We are the connecting tissue across the various core capabilities offered by our sibling teams. Joining the team means working on fundamental problems in event‑driven orchestration systems using open‑source and in‑house technologies. Your focus will include python applications, performant queue brokers in Go, C‑based Python extensions and more. If you enjoy distributed systems or orchestration platforms, this is the place for you.

• Develop, deploy and integrate an internal priority queue system to mitigate head‑of‑line blocking and optimize time to market for key tasks
• Deprecate the on‑cloud orchestration system by unifying with on‑prem system
• Enable component registration/management to simplify onboarding for service owners
• Introduce multi‑tenancy with granular privileges for both pipeline creation/edit and runtime
• Enhance support and capability for DAG creation in orchestration
• Programmatic recipe creation, modification and interaction to be API‑first
• Integrate LLM to assist in authoring ETL tasks

• Design user‑friendly, fault‑tolerant and scalable orchestration platform components
• Participate in the full SDLC of platform components and systems
• Familiarize yourself with data processing and analysis tasks taken on by the Data Department and systems built by sister teams
• Bring the latest OSS developments and capabilities into our stack
• Mentor less‑experienced team members

• Demonstrable experience building fault‑tolerant and horizontally scalable distributed applications or systems, addressing distributed‑system‑specific concerns
• Experience writing production code in Python, Go, Rust, JavaScript/Type Script, or Java
• Strong problem‑solving skills

• Knowledge of database systems and trade‑offs in distributed systems
• Familiarity with API design
• Experience with orchestration frameworks such as Apache Airflow, Argo Workflows, Dagster, etc.
• Experience designing and using systems on AWS

When indicated, we will consider applications from all candidates who can demonstrate the required skills; years of experience are only a guide.