Senior Data Platform Engineer

Position: Atcon Global - Senior Data Platform Engineer
About the role

You support the design of client's multi-user data and analysis platform, in particular the Storage layer component (including the data lake, which forms the basis of the data platform). You also support interactions with other components ('ingestion layer', 'core systems', 'front-end-systems').

You specialize in data and analytics platforms, with expertise in lakehouse architectures, Kubernetes-based data science environments, open source ecosystems, and European digital sovereignty initiatives.

You have experience in designing scalable, interoperable, secure (GDPR, NIS2) and production-oriented platforms using technologies such as PostgreSQL, MinIO, Open Stack, Apache, Kubernetes, Docker, Spark, Onyxia, and modern governance and metadata frameworks.

You will be the link between the architecture at the organizational level and the detailed technical implementation. You collaborate with internal departments, technical teams and external partners to find an integrated and robust solution.

Responsibilities

Propose, evaluate, test and provide advice on technologies, tools and methodologies for the 'Storage' component (lakehouse). This includes

Blob Storage, data lake

Data Warehouse relational database management system (RDBMS)

SQL RDBMS

No

SQL Database System.

Propose how to implement the 'Medallion' architecture (bronze, silver, gold), archiving, and back-up within the data platform, taking into account the ingestion, processing, integration, consumption and dissemination of the different types of data flows and the obligations (legal, Belgian health data ecosystem, animals and environment).

Evaluate, propose and test that these solutions are consistent with the general architecture of the platform, are based on open source, sovereign and multi-user technologies, and are compatible and interoperable with the other 'Application architecture' components, in particular with:

'ingestion layer' (including the different sources)

'core systems'

'front-end-systems' (data science workbench, BI, data service Gate Way, secure processing environment)

Ensure that the architecture and data flows comply with the requirements defined by the Data Office, GDPR, NIS2, Information Security Committee (ISC), the new registry policy and the European Health Data Space (health data holder, trusted data holder).

Proactively solve challenges related to system choices and implementation

Advise technical teams, evaluate technical designs and code, support best practices;

Collaborate with the various departments involved, management and external partners and act as a technical contact person;

Write (technical) documentation

Ensure high-quality technical implementation

Technical requirements

At least 10 years of experience in data platforms, data engineering, solution architecture, data Storage, Big Data ecosystems, data modeling, and database technologies;

Design and implementation of modern data and analytics platforms in lakehouse architectures, open source ecosystems (without single vendor lock in), and European digital sovereignty initiatives.

Data Platform Architecture : data ingestion, data lakes, data warehousing, ETL/ELT pipeline

Technical expertise in the selection, implementation and maintenance of:

Stockage (e.g. MinIO, PostgreSQL, No

SQL, S3-compatible object Storage)

Data processing (e.g. Apache ecossystem, Iceberg, Spark, DBT,)

Containers and orchestration (e.g. Docker, Kubernetes, Helm)

Securité (e.g. Keycloak, Vault)

Infrastructure (e.g. Terra Form, Open Stack)

Géstion et Version Control (e.g. open project, Jira, Git Hub, Git Lab)

Stack cloud souveraine (e.g. Ovhcloud, Hetzner, Scaleway)

Metadata management, data governance, privacy, data protection and security (e.g. GDPR, NIS2, Data Governance act)

Language requirement:
Dutch or French; and excellent knowledge of English

Behavioral skills

Strong analytical and problem-solving skills

Excellent interpersonal and communication skills, able to convey complex topics to a technical and non-technical audience

Self-motivated and able to work independently, with a collaborative mindset and strong communication skills

Ability to lead, help define and influence the design and development of solutions

A customer-oriented mindset with a strong results-oriented approach