Senior Data Engineer

Responsibilities

• Design, build, and improve critical batch and near‑real‑time data pipelines that support enterprise analytics and operational use cases.
• Develop reusable engineering patterns for ingestion, transformation, storage, and serving layers across the platform.
• Own important platform components and ensure they are reliable, scalable, and supportable in production.
• Contribute directly to the modernisation of legacy data workflows into stronger platform‑aligned solutions.

• Contribute to the design and evolution of the enterprise data lake or lakehouse platform.
• Implement and refine engineering standards for storage layout, transformation patterns, and data processing frameworks.
• Optimise partitioning, schema evolution, and file organisation to improve performance and maintainability.
• Build and support distributed data processing solutions using modern frameworks and platform tooling.

• Design and implement data quality, reconciliation, and observability controls for critical datasets and platform flows.
• Ensure key pipelines and datasets meet expectations for freshness, completeness, accuracy, and recoverability.
• Strengthen metadata, lineage, and documentation practices across the platform.
• Work with governance and security stakeholders to support compliant, well‑controlled data management practices.

• Partner with BI, analytics, software engineering, product, and business stakeholders to understand and support data use cases.
• Translate business and platform requirements into scalable, well‑engineered technical solutions.
• Provide hands‑on technical leadership during planning, design, implementation, and operational improvement.
• Mentor intermediate and junior data engineers through code reviews, design guidance, and practical knowledge‑sharing.

• Identify opportunities to improve platform performance, resilience, scalability, and cost efficiency.
• Drive automation, standardisation, and maintainability across the data platform.
• Evaluate new tools or patterns where they provide clear value to the platform or team.
• Help grow the maturity of BET Software's data engineering capability over time.

• The platform may include a combination of established and modern technologies such as: SQL Server, Python, Spark, Flink, Airflow, Object storage, Open format Tables, Kafka / Redpanda, Click House or similar columnar analytical stores, Git and CI/CD tooling, Kubernetes, Open Shift, or similar runtime environments.

• Degree or diploma in IT, Computer Science, Engineering, or a related technical discipline.
• 6+ years of experience in data engineering, ETL/ELT development, or data platform engineering.
• Strong hands‑on SQL expertise, including advanced performance tuning, query optimisation, indexing strategies, and efficient analytical data design.
• Proven experience building and operating modern data platform components in production environments.
• Strong experience working with object storage in cloud or on‑premises environments.
• Experience with workflow orchestration platforms such as Airflow, SQL Server Agent, or similar.

• Data Warehouse, Lake and Lakehouse architecture patterns.
• Distributed data processing using frameworks such as Spark or Flink.
• Designing and supporting batch and near‑real‑time ingestion pipelines.
• Building incremental, idempotent, and fault‑tolerant data pipelines.
• Data quality, reconciliation, and observability practices.
• Metadata, lineage, governance, and access control concepts.
• Analytical data modelling and efficient data structures for warehouse and large‑scale query workloads.

• Medallion architecture such as Bronze, Silver, and Gold layers.
• Open table formats such as Iceberg.
• Schema evolution, partitioning strategies, file optimisation, and storage layout tuning.
• Event‑driven or streaming platforms such as Kafka, Pulsar, or Redpanda.
• Columnar or high‑performance analytical platforms such as Click House.
• CI/CD pipelines, deployment automation, and engineering…