A Unified and Native Language Understanding Benchmark for Turkic Languages

View a PDF file from the paper entitled TUMLU: standards for understanding the unified and united language for Turkish languages, by Jafar ISBAROV and 15 other authors

PDF HTML (experimental) view

a summary:The ability to evaluate the possibilities of understanding the massive multi -tasking language (MMLU) in a comprehensive way is necessary to apply the application of multi -language language models. However, preparing such standards in the high -quality mother tongue is often expensive, and thus limits the representation of evaluation data sets. Although modern efforts that focused on building MMLU standards are more comprehensive, they were traditionally designed using automatic translation of high resource languages, which may provide errors and fail to calculate the linguistic and cultural complications of targeted languages. In this paper, we address the lack of a mmlu standard in the original language, especially in the Turkish language family, an imperfect representation with the characteristics of Morvostine distinctive. We suggest the criteria of the Turkish language MMLU: Tumlu is comprehensive and multi -language standards, and an original advanced language understands specially designed for Turkish languages. It consists of questions at the level of middle and secondary schools that extend to 11 academic topics in Azerbaijani, Crimea Tatars, Karaclac, Kazakh, Tatar, Turkish, Ouigor, and Uzbek. We also offer TUMLU-MINI, a more brief, balanced and manual sub-set of the data set. Using this data collection, we evaluate a variety of large open and ownership models (LLMS) systematically, including Claude, Gemini, GPT, and Llama, as it provides an in -depth analysis of its performance across languages, topics and various edities. To enhance more research and development in the understanding of a multi-language language, we launch Tumlu-Mini and all the corresponding evaluation programs.