Data Engineer AI; f​/m​/d

In the role of Data Engineer for AI (f/m/d), you will design, build and operate data products and reusable data preparation components on the Data and AI Platform at Allianz Global Investors, built on Databricks and Foundry.

Your focus is to enable platform customers to reliably source, ingest, transform, validate and serve high-quality, compliant data for AI and ML use cases (including analytics and GenAI), so teams can consume data in a self-service manner. You will provide technical guidance and L3 support to delivery teams, define and promote best practices for data pipelines and data quality, and ensure adherence to security, privacy and regulatory requirements following a compliance-by-design approach.

In addition, you will continuously evolve the platform’s data engineering capabilities by integrating new features (e.g., ingestion patterns, transformation frameworks, governance controls and monitoring) and by delivering standardized, reusable pipelines and templates that scale across use cases. Databricks is the platform’s primary technology for data engineering, with Microsoft Foundry part of its wider scope.

This position will be based in Frankfurt.

• Reusable data preparation building blocks & ML-ready pipelines:
  Develop, maintain and enhance standardized pipeline patterns, templates and utilities (ingestion, transformation, validation, enrichment) to deliver AI/ML-ready datasets. Build robust pipelines for structured and unstructured data with reproducible, deterministic outputs
• Data pipeline engineering on Databricks:
  Build and operate reliable batch and streaming pipelines to ingest data from internal and external sources, curate datasets, and publish trusted data products for AI, ML and analytical consumption
• Curated data products (Bronze/Silver/Gold):
  Establish controlled dataset evolution, enforce data quality and freshness, and curate gold tables explicitly designed for analytical, ML and AI use cases.
• Feature engineering & feature store:
  Design, build and operate feature engineering pipelines and curate a ready-to-use feature store with proper versioning and lightweight documentation (ownership, purpose, inputs/outputs, SLAs)
• GenAI / RAG data & embedding pipelines:
  Collaborate with ML and AI Engineers to build and operate pipelines for GenAI/RAG systems, including new data ingestion, Bronze/Silver updates, chunking, embedding refresh and index updates to maintain knowledge base relevance
• Systematic experiments & model improvement loops:
  Enable systematic experimentation (offline evaluations, A/B tests) and support model retraining using user feedback (binary and non-binary) to continuously improve models
• Platform capability evolution (Data for AI):
  Collaborate with the Data and AI Platform Architect to introduce and operationalize platform capabilities for ingestion, transformation and serving (Delta/Unity Catalog patterns, orchestration, monitoring), making them available securely and consistently
• New data source onboarding & Lakeflow connectors:
  Facilitate provisioning and rollout of Databricks Lakeflow connectors to integrate new data sources and standardize ingestion across domains
• Security, governance & compliance-by-design:
  Implement and enforce governance using Unity Catalog, including access control, lineage, retention, encryption and auditing, ensuring compliance with internal standards and regulations
• Data quality, validation & reliability:
  Establish automated data quality checks and validations (e.g., DQX), define SLAs/SLOs for freshness and reliability, and implement monitoring and observability
• Performance & cost optimisation:
  Optimize Spark workloads, storage layouts and compute usage to ensure scalable, stable and cost-efficient data pipelines
• Colla…