Hello everybody!

Today I will be talking about a research work that we started with the wonderful Dieuwke Hupkes approximately 3 years ago, officially released more than an year ago, and finally got accepted about a month ago! Research is not for the impatient!

What is MultiLoKo

As the tittle suggests, MultiLoKo is a multilingual knowledge benchmark for LLMs. What makes it special is that it aims to test LLMs' multilingual capabilities in realistic usecases. For the impatient ones, here is a quick tldr of this blog post.

• Local knowledge benchmark on 31 languages.
• Frontier models from last year barely score 30% on it.
• Frontier models from this year manage about 50%.
• Small local models are unsurprisingly quite bad on the task.
• Notably different result from simple translated benchmarks such as multilingual MMLU.

The problem with multilingualism in LLMs

What gets measured, improves. The LLM world consists of loud minority of expert users who care about code/reasoning/math in English and Chinese (as a by product of what countries the largest LLM players come from), and a silent majority of casual users in many different countries. The most popular benchmarks (and this was especially true in 2023) are catered towards English, coding, math, reasoning.

Multilinguality is an afterthought: Models need to be multilingual, but since the most important benchmarks are not multilingual, it's not prioritised as much as it should be.

The most famous multilingual benchmark remains Multilingual-MMLU and that was basically the only widely used multilingual benchmark back in 2023. The issue with this benchmark is that it is translated.

The translation issue of Multilingual benchmarks

Since machine translation is cheap and widely available, a stop gap solution to evaluating multilingualism in LLMs is to just translate English benchmarks. This works somewhat ok for math and logic, but it completely misses the point in knowledge tasks:

• A Bulgarian native speaker is far more likely to ask an LLM about the Bulgarian national hero Vasil Levski as opposed to say Robin Hood
• A translated English centric knowledge benchmark is far more likely to ask about Robin Hood, and I bet my hat that an English centric knowledge benchmark won't contain information about Vasil Levski.

When LLM creators make the complex decisions of which datasets are used during training, they look at benchmarks as a guidance. And while intuitively, adding a whole lot of Bulgarian historical documents (for example) might feel like a right step towards multilinguality, without a benchmark it's difficult to make that decision:

• How do you know if the data is high quality? I am a native Bulgarian speaker, but many people aren't and can't browse through the data easily.
• How do you know if the data you add actually improves any features you care about?

In short, we needed a multilingual knowledge benchmark that targets locally relevant knowledge, and that's why we created MultiLoko!

• 31 Languages
• 500 questions for each Language, split into dev and a secret, hidden test set.
• Machine and Human Translations of all non-English questions to English and of all English-questions to all 31 languages, so that we can measure cross-lingual knowledge transfer.

Benchmark design

For the benchmark, we wanted to make it difficult, so that it has some longevity, but at the same time not incredibly unlikely and out of domain so that even humans wouldn't have a clue about the questions. We decided the best way to do that is to go to wikipedia.

Data selection

The wikipedia dataset is contained in the training of every single LLM, so that guarantees that the questions we form would definitely be something that the LLM ought to get right. Wikipedia also provides us with various statistics of the individual web pages, which allows us to select suitable data:

• Take the top 3500 most visited wikipedia pages for all 31 languages over a 10 year period.
• For each non-English language remove from the selection pool pages where the topic is very famous (we judge this by using the information "How many languages has this topic been translated into?" from the wikipedia statistics). Thus questions about Lionel Messi or Christiano Ronaldo are not part of the dataset.
• Sample a paragraph of 500 words or so to give to Annotators

Annotator instructions

We went through many iterations of the annotation protocol and we settled on the following.

• Annotators are native speakers of each of the 31 language and go through a lengthy explanation with examples of what good locally relevant questions look like. Basically the same explanation that you just read in the few paragraphs above.
• Annotators receive the list of pre-filtered topics and paragraphs and select 500 of them to formulate a question and an aswer pair. All possible surface forms of the correct answer are produced (IE one;1;first).
• Each question produced is audited by an independent annotator. They are just given the source paragraph and the question and tasked to produce the answer. If they fail to produce the same answer, the question is discarded and a new one has to be produced.
• Finally an independent validator goes over the questions produced and marks them as suitable or not.

Examples

All questions are indended to be answered with one word or short phrase. We wanted to avoid MCQ style questions because they are game-able. Evaluation is done on ExactMatch basis. We also experimented with soft metrics such as chrf or includes, and even did LLM-as-a-Judge, but they didn't show a different trend from ExactMatch and we decided to stick to that for simpliciity.

• How old was Prince Phillip when he married Princess Elizabeth?
• De todas las hermanas y medias hermanas de Frida Kahlo, ¿quién fue la mayor de todas?
• 张无忌在哪座山修练猴腹之中的九阳神功？

Findings

We will present basic findings here, but for more details please check the paper.

Overall results

Back when we released the benchmark, none of the models really cracked the 50% mark, indicating that even though the information was there, the task reamined relatively difficult:

Translation results

Since we commissioned translation, we are able to check models' abilities to do cross lingual knowledge transfer!

Mother Tongue Effect

The mother tongue effect is the difference between asking a question about Bulgarian history in Bulgarian, versus asking the same question in English. Positive mother tongue effect means native language is better, negative means English is better.

For language where the model is weak, translating the question in English helps understanding and therefore we see negative mother tongue effect.

Locality Effect

The locality effect is the opposity. Here we take the English dataset and translate it in every other language, mimicing the most popular setup in machine translated datasets. Positive locality effect means we underestimate model performance by asking English centric questions, negative locality effect means we overestimate model performance by asking English centric questions.

We see that if we had used a translated dataset we would have gotten a very different conclusion about a lot of the languages.

Consistency

Ultimately, we care about consistency. In a perfect cross-lingual transfer world, a model should produce exactly the same answer to an identical question asked in English and non-English regardless of whether that answer is right or wrong.

We can see that asking the same question in English and non-English produces wildly different results, especially as we go towards the tail of the languages, indicating there is plenty of work to be done in the field of cross-lingual knowledge transfer.

Conclusions

A quick tl;dr of the results is:

• The benchmark is hard, none of the models perform that well on it.
• Cross lingual transfer is not a solved problem, your mileage will vary based on the language pair.
• Using Machine Translated knowledge datasets is not a good way to predict performance in a given language, because it is unlikely for a Bulgarian speaker to ask the model about the climate of Alabama, and we should really not evaluate multilingual performance on English centric knowledge.

As we said earlier, what gets measured, improves. Now that we have a good multilingual dataset, we can do a better job at selecting multilingual datasets for pretraining!

Related Work

Back when we started this work in 2023, there were no similar benchmarks. All multilingual knowledge evaluation sets were strictly of the translated variety such as Multilingual-MMLU. In the meantime though, a number of similar works have popped up.

• INCLUDE 44 languages, MMLU style categories. Multiple-Choice-Questions.
• Global-MMLU 42 languages, MMLU style categories. Multiple-Choice-Questions
• M3Exam 9 languages, state examinations. Multiple-Choice-Questions

It is important to note that the role of those test sets is complementary as they use different sources to formulate their questions. MultiLoko is the only generation benchmark out of the bunch.

How do 2026 models perform

Since it's been a few years since the dataset was released, time to check how modern models perform on it!

Small local models

Projects such as llama.cpp or Ollama have popularised running small models on local machines, for privacy, security or monetary reasons. Those models allegedly make good personal assistants and coding companions, but they don't really do that well on knowledge tasks:

It is pretty clear that model size is the most important thing: The bigger the model, the better the quality, with dense models seeming to have a tiny edge. Also the gemma family performs better than the Qwen family.

Finally, we noticed some instruction following issues and decided to include the "contains" in order to make sure we are not unfairly penalising the model for being too verbose. Even with this in mind, the results are pretty lackluster.

Frontier models

Let's see if the frontier proprietary models do any better. Here, we decided to include an extra metric Lenient EM which basically involves an LLM trying to extract the answer from output. This aims to abstract away from instruction following issues and focus on pure model capabilities.

We see several very interesting trends here:

• Model performance has improved by up to 20% on average from gemini and gpt. Quite impressive, although still far from perfect.
• Gemini 3.5 flash's performance gap between EM and Lenient EM is nearly double that of GPT 5.5. This suggests that Gemini is worse at following isntructions than GPT. In practise this results in more verbose answers that users will have to read through in order to get to the answer. It is interesting that the gap between the two models disappears when using lenient metrics, suggesting they are quite similar.
• Claude is known for focussing on coding, and its score virtually not moved from the 3.5 release. This suggests that Anthropic don't care that much about increasing their knowledge capabilities, and are single mindedly pursing coding, unlike their competitors.

Frontier models on hidden test set

We mentioned earlier that we split our dataset into dev and test set, and the test set is not publicly available online. This dataset is slightly harder and frontier labs don't have the opportunity to overfit on it:

We can see that there is a meaningful difference across the board between the dev and the test set! Models lose about 5% of their performance when evaluated on unseen questions, suggesting at least some degree of overfitting.

Conclusions

A quick tl;dr of the results is:

• The benchmark is hard, none of the models perform that well on it.
• Cross lingual transfer is not a solved problem, your mileage will vary based on the language pair.
• Using Machine Translated knowledge datasets is not a good way to predict performance in a given language, because it is unlikely for a Bulgarian speaker to ask the model about the climate of Alabama, and we should really not evaluate multilingual performance on English centric knowledge.
• Over the past year, frontier models have gained up to 20% on the benchmark, but they are still quite far away from acing it.
• Anthropic seem to focus on their coding capability the most, ignoring knowledge. Their scores basically have not moved over the past year.
• Small local models do quite poorly on the benchmark, model size is quite important when it comes to the knowledge capability.

As we said earlier, what gets measured, improves. Now that we have a good multilingual dataset, we can do a better job at selecting multilingual datasets for pretraining!

Related Work

Back when we started this work in 2023, there were no similar benchmarks. All multilingual knowledge evaluation sets were strictly of the translated variety such as Multilingual-MMLU. In the meantime though, a number of similar works have popped up.

• INCLUDE 44 languages, MMLU style categories. Multiple-Choice-Questions.
• Global-MMLU 42 languages, MMLU style categories. Multiple-Choice-Questions
• M3Exam 9 languages, state examinations. Multiple-Choice-Questions

It is important to note that the role of those test sets is complementary as they use different sources to formulate their questions. MultiLoko is the only generation benchmark out of the bunch.

Thank you for reading so far! For mode details please make sure to check the paper! Make multilingualism better and remember to have fun while doing research!