December 11 in Astana to the President Kazakhstan's Kassym-Jomart Tokayev was presented with the national language model KAZ-LLM. The model was developed under the guidance of the Institute of Smart Systems and Artificial Intelligence (ISSAI NU) in partnership with Beeline Kazakhstan and its IT company QazCode, as well as Astana Hub. The project is coordinated by the Ministry of Digital Development, Innovation and Aerospace Industry of the Republic of Kazakhstan (ICRIAP RK).  The model is of strategic importance for the whole country, as it solves the problem of the language gap with the help of AI.

How was the KAZ-LLM model developed?

ISSAI's KAZ-LLM is based on 150 billion tokens, carefully collected from publicly available sources in four languages — Kazakh, Russian, English and Turkish. This allows the model to demonstrate high accuracy and versatility, providing improved text processing quality in various languages and contributing to improved translation. Tokens are the minimum units of text, such as words, parts of them, or even individual characters that the AI uses to analyze and understand information. 

The interface and functionality of the KAZ-LLM model have been developed taking into account the most advanced international standards, which confirms the high technological maturity and wide potential of the model. To assess its performance, comprehensive benchmarks with question-answer pairs covering a variety of fields of knowledge were used. The benchmark package included the following tests:

• ARC (AI2 Reasoning Challenge) is a test of scientific thinking through multiple choice questions.
• GSM8K — assessment of the ability to solve math problems for elementary school.
• HellaSwag — testing the logic of sentence continuation.
• MMLU (Massive Multitask Language Understanding) is a knowledge test in 57 different subjects.
• Winogrande — evaluation of common sense in ambiguous offers.
• DROP — testing of reading comprehension and logical thinking skills.

Beeline and QazCode partnership accelerated the development

Beeline Kazakhstan and its IT company QazCode became key partners in its creation, combining efforts and experience in creating language models such as Kaz-RoBERTA, as well as in the development of AI solutions for small language groups in cooperation with foreign partners. Support in the form of provided servers with computing power of 8 DGX H100 significantly accelerated the learning process and expanded the capabilities of the model. For comparison, an ordinary computer will take several days to analyze an archive of 1 million photos. While 8 DGX H100 servers used for ISSAI KAZ-LLM training will cope with this task in just a few seconds.

On the basis of these servers, the developers trained two versions of the model — with 8 billion and 70 billion parameters, and QazCode data scientists joined the process. 

“Our team actively participated in the development and training of the KAZ-LLM model. When creating LLM, developers and partners used modern machine learning technologies such as PyTorch and Torchtune, and also took into account the experience of previous projects on adapting open source LLM architectures for the Kazakh language. During the training, which lasted 50 days of continuous computing, the model improved the ability to understand the context and ensure high-quality user interaction. Testing has shown that the model successfully solves technical problems, taking into account the cultural and linguistic features of the Kazakh language," said Alexey, CEO of QazCode Sharavar.

About the results and prospects of KAZ-LLM

The researchers note that the project is an important milestone on Kazakhstan's path on the world stage of artificial intelligence: "This model reflects Kazakhstan's desire for innovation, independence and growth of its technological ecosystem. Our team has prepared two versions of ISSAI KAZ-LLM with 8 billion and 70 billion parameters, built on the architecture Meta Llama and optimized for high-performance systems and environments with limited resources. The models are released under a CC-BY-NC license, which are available for non-commercial use on the Hugging Face website, facilitating global academic and research collaboration. In this way , developers will be able to download and run our model on both complex servers and laptops," said Professor Hussein Atakan, Director of ISSAI Varol.

ISSAI Kaz-LLM is expected to open up new opportunities for creating startups and innovative AI-based projects.  In the future , it is planned to develop next-generation models that will integrate language and visual data, which will significantly expand the capabilities of AI. The addition of support for the model of other Turkic languages is also being considered, which will strengthen ties between Turkic-speaking communities.