Data Engineer; m​/f​/d

Stellenbezeichnung: Data Engineer (m/f/d)
.

-.: DE20971

Schedule type:
Full Time Contract Type:
Permanent Position WHAT WE ARE LOOKING FORDesign, further development, and practical implementation of a scalable data lakehouse and data platform architecture for very large, heterogeneous data sets Developing a long-term data and storage strategy that takes into account performance, scalability, data lifecycle, costs, availability, and operational requirements

Designing suitable storage and access concepts for large volumes of image, video, sensor, geospatial, and other unstructured and semi-structured data, including metadata, partitioning, indexing, compression, and archiving

Building and operating robust batch and streaming pipelines for the ingestion, processing, transformation, and delivery of data from heterogeneous sources

Co-design and implementation of a semantic data model or ontology that enables data objects, entities, relationships, and context to be described and linked across systems

Creating a resilient data foundation for machine learning, AI-based applications, and autonomous systems, including traceable data provenance and reproducible training and validation datasets

Ensuring data lineage, data versioning, data quality, schema management, and appropriate governance and access concepts throughout the entire data lifecycle

Development and operation of the data platform in large, distributed Kubernetes-based infrastructures, as well as integration into Infrastructure-as-Code and CI/CD/Dev Sec Ops  processes

Analyzing and optimizing data layouts, data locality, compute proximity, and data movements for data-intensive applications

Evaluation of suitable open-source and commercial technologies, as well as implementation ranging from proofs of concept to production-ready, scalable solutions

Close collaboration with AI/ML, software, platform, and cybersecurity teams on architectural decisions, interfaces, data products, and operational implementationWHAT QUALIFICATIONS YOU SHOULD HAVEA degree in computer science, data engineering, software engineering, business informatics, or a comparable practical qualification

Several years of hands-on experience in data engineering and in building or further developing distributed data platformsA very good understanding of modern data lake/lakehouse architectures, object storage, and the fundamental trade-offs between storage, compute, data movement, performance, and costs

Demonstrable hands-on experience with data pipelines, data modeling, SQL, and at least one relevant programming language, preferably Python Solid knowledge of metadata, data quality, schema management, data lineage, and reproducible processing

Practical understanding of Linux, containers, and Kubernetes, as well as the ability to independently implement, operate, and analyze data components in a distributed platform

Ability to link architectural decisions to practical implementation and to systematically investigate problems down to their technical root causesA structured, team-oriented approach to work, as well as strong written and spoken English skills

Experience with very large volumes of image, video, sensor, or geodata

Knowledge of ontologies, knowledge graphs, semantic data models, entity resolution, or comparable approaches

Experience with technologies for distributed data processing and streaming, e.g., Spark, Flink, Trino, Kafka, or comparable solutions

Experience with open Lakehouse table formats or comparable technologies, such as Apache Iceberg, Delta Lake, or Apache Hudi Experience in provisioning and versioning data for machine learning, MLOps, AI, or autonomy workloads

Knowledge of Infrastructure as Code, Git Ops, or automated platform and deployment processes

Experience with highly available, on-premises, edge,…