Senior Data Engineer – Data Products & AI

Beacon Intelligence delivers data and insights that help R&D scientists develop better pharmaceuticals, faster than ever before.

Our focus is simple: we curate high-accuracy data enriched and transformed with AI. This is delivered through a range of platforms, increasingly via AI-native products and features that enable customers to extract insights faster than ever before.

This is a rare opportunity to make a meaningful impact on patients’ lives worldwide.

As a Senior Data/AI Engineer, you will play a central role in building and scaling our data products.

This is not a traditional back-end data engineering role. You will work closely with product, commercial and technical colleagues, transforming datasets into commercial, customer-facing solutions.

You will combine deep technical expertise with a strong understanding of how data can be structured, enriched and ope rationalised through AI systems.

• Design and build scalable data products underpinning Beacon’s commercial offerings
• Transform raw and third-party data into structured, enriched, product-ready datasets
• Partner with product and commercial teams to define how data is packaged, accessed and monetised
• Enable delivery via APIs, internal tools and customer-facing platforms
• Apply AI and LLM capabilities to enrich and enhance data (e.g. classification, tagging, summarisation, insight generation)
• Design and build LLM-powered product features
• Integrate RAG into pipelines to improve data quality and unlock new features
• Support development of AI-driven products such as recommendation engines, search and insight tools
• Ensure systems are optimised, well-governed and resilient
• Support deployment and lifecycle management of data and AI systems
• Own and evolve the data platform architecture (Databricks, Azure, Airbyte) to support scalability
• Own performance and cost optimisation across Azure Databricks and supporting Azure services, including compute selection, autoscaling, workload monitoring and resource-efficiency improvements.
• Build and maintain robust data pipelines (batch and streaming) delivering reliable, production-ready datasets
• Ensure high standards in data quality, testing and observability
• Improve efficiency through automation, CI/CD and engineering best practice

• Partner with senior stakeholders to identify high-value data product opportunities
• Translate business needs into practical, scalable data and AI solutions
• Act as a bridge between technical teams and commercial/product stakeholders
• Contribute to prioritisation based on business impact

• Shape the evolution of our data product and AI strategy
• Mentor junior engineers and promote a strong engineering and product mindset
• Help build a high-performing, commercially aware data and AI engineering team

• Deep, hands‑on experience designing, building and operating production data platforms using Azure Databricks and Azure Cloud environments
• Strong experience with Apache Spark, PySpark, Spark SQL and Delta Lake, including incremental processing, schema evolution, performance optimisation and batch and streaming workloads.
• Strong experience implementing Unity Catalog, including catalog and schema design, access control, data lineage, discovery and governance of production data assets.
• Proven experience designing lakehouse and Medallion architectures and converting raw and third‑party data into governed, reusable data products.
• Advanced Python and SQL, focused on production‑quality, scalable solutions
• Proven experience designing and building data pipelines and models
• Experience working with APIs and data delivery mechanisms (critical for productisation)
• Ex…