Internship | Cross-lingual transfer Dutch LLM pretraining

Data scarcity is a major challenge for developing European sovereign foundation models, especially for low-resource languages where additional training data is inherently limited. A promising solution is cross-lingual transfer: training models on related languages can improve performance on a target language more effectively than adding more data from the target language itself. The recent ATLAS study (Longpre et al., 2025) is the largest empirical investigation of this effect to date, training 774 multilingual models across 50 languages and introducing the Bilingual Transfer Score (BTS) to measure how much one language benefits another.

The study found that shared language family and script are the strongest predictors of transfer effectiveness. However, Dutch was not included in the ATLAS language set. This creates both a scientific and practical gap. ATLAS's predictors cannot distinguish whether English or German should be a better source language for Dutch, as both are Germanic languages written in the Latin script.

Yet, in practice, one may provide substantially more benefit than the other. Answering this question is directly relevant for Dutch and European sovereign LLM initiatives, which must decide how to balance Dutch, English and other related languages within pretraining datasets to maximize performance.

The goal of this project is to extend ATLAS's methodology to Dutch and investigate:

Whether Dutch benefits more from English or German during pretraining, and whether this can be explained by ATLAS's language-family and script-based predictors. Which Dutch-English-German data mixture yields the best combined Dutch and English performance under a fixed compute budget.

Phase 1:
Extending Bilingual Transfer Scores to Dutch
Study and reproduce ATLAS's BTS methodology. Train a Dutch-only baseline model and several bilingual variants, including Dutch+English and Dutch+German, with one or two additional control languages. Compute BTS scores relative to Dutch. Compare transfer effectiveness across source languages and assess whether ATLAS's existing language-family and script categories sufficiently explain the results.
Phase 2:
Optimizing multilingual data mixtures
Run a small experimental sweep over Dutch-English-German data mixtures. Evaluate each model on Dutch and English test sets using ATLAS-style metrics. Compare the resulting performance against simple baselines such as Dutch-only and equal-split mixtures. Analyze whether optimal mixture ratios can be predicted from pairwise transfer scores or require direct experimentation.

• A reproducible Dutch-focused replication of the ATLAS BTS methodology.
• An empirical comparison of English-to-Dutch versus German-to-Dutch transfer.
• A recommended Dutch-English-German pretraining mixture for sovereign LLM development.
• A thesis discussing the methodology, findings, and limitations of scaling conclusions from small proxy models.

This assignment is aimed at students of the MSc Artificial Intelligence or adjacent fields. There are supervisor meetings once a week, possibly combined with other students to share knowledge across projects.

An internship at TNO means working in an environment where substance and impact are central. You will become part of a knowledge organization where research and practice come together, and where experts collaborate on solutions to current societal and technological challenges. Your internship is a period in which you can discover what suits you, where your strengths lie and what you would like to learn next.

You are part of a professional working environment, gain insight into how things work in practice, and have the opportunity to build experience that goes beyond this internship alone. For many students, an internship is therefore also a first step in discovering whether TNO could be a potential next step after graduation.

• A professional and innovative internship environment in which you actively contribute to societal and technological challenges, working alongside leading experts in your field.
• Personal and dedicated supervision, with focus on your…