kaz29Вестник Казахстанско-Британского технического университетаHerald of the Kazakh-British Technical University1998-66882959-8109Казахстанско-Британский Технический Университет10.55452/1998-6688-2026-23-3-336-346kaz29-3196Research ArticleКОМПЬЮТЕРНЫЕ НАУКИCOMPUTER SCIENCEБЕНЧМАРКИНГ МОДЕЛЕЙ ГЛУБОКОГО ОБУЧЕНИЯ ДЛЯ КЛАССИФИКАЦИИ КАЗАХСКИХ НОВОСТНЫХ ТЕКСТОВ В УСЛОВИЯХ ОГРАНИЧЕННОЙ РАЗМЕТКИBENCHMARKING DEEP LEARNING MODELS FOR FEW-SHOT CLASSIFICATION OF KAZAKH NEWS TEXTS UNDER LIMITED ANNOTATIONhttps://orcid.org/0009-0005-3266-2126МарламбековД.MarlambekovD.

Докторант

Алматы

PhD-student

Almaty

dmarlambekov@gmail.comhttps://orcid.org/0000-0002-7718-6478АхметоваA.AkhmetovaA.

Докторант

Алматы

PhD-student

Almaty

baltabekova1994@gmail.comhttps://orcid.org/0009-0003-8161-533XТөрекулС.TorekulS.

Докторант

Алматы

PhD-studentAlmaty

saule.torekul@gmail.comhttps://orcid.org/0000-0003-3853-8896УалиеваИ. М.UaliyevaM.

К.ф.-м.н., ассоциированный профессор

Алматы

Cand. Phys.-Math. Sc., Associate Professor

Almaty

i.ualiyeva@gmail.comКазахский национальный университет им. аль-ФарабиКазахстанAl-Farabi Kazakh National UniversityKazakhstan202626092026233336346Copyright © Марламбеков Д., Ахметова A., Төрекул С., Уалиева И.М., 20262026Марламбеков Д., Ахметова A., Төрекул С., Уалиева И.М.Marlambekov D., Akhmetova A., Torekul S., Ualiyeva M.Данная работа распространяется под лицензией Creative Commons Attribution 4.0.This work is licensed under a Creative Commons Attribution 4.0 License.https://vestnik.kbtu.edu.kz/jour/article/view/3196

Стремительное развитие методов глубокого обучения и появление трансформерных архитектур кардинально изменили ландшафт обработки естественного языка, однако эти достижения остаются неравномерно распределенными. В то время как для английского и других высокоресурсных языков существуют масштабные бенчмарки, для низкоресурсных языков, таких как казахский, проблема качественной классификации текстов остается острым вызовом. Авторами проведен сравнительный анализ классической модели эмбеддингов Word2Vec, рекуррентных моделей BiLSTM, трансформеров BERT и XLM-R, генеративной модели mT5, а также классических и статистических бейзлайнов для задачи few-shot классификации казахских новостных текстов в условиях ограниченной разметки. Эксперименты проведены на датасете KazNews для few-shot классификации, собранном из статей tengrinews.kz и inform.kz и параллельных корпусов HuffPost. Датасеты содержат пять категорий (спорт, политика, бизнес, путешествия и др.). Эксперименты в режимах 1-shot и 5-shot продемонстрировали, что ни одна из моделей не доминирует одновременно на всех датасетах и режимах: Word2Vec показал лучшие результаты на датасетах на корпусах HuffPost (0.48 и 0.53) при 5-shot режиме; mT5-Seq2Seq лучший на KazNews при 5-shot (0.71); BERT в целом конкурентоспособен при 5-shot на датасете KazNews с длинными текстами. Авторы также выявили резкий скачок качества для модели mT5-Seq2Seq в 5-shot. Для режима 1-shot ни одна из моделей не показала приемлемого результата качества.

Deep learning methods and transformer architectures have fundamentally reshaped the field of natural language processing; however, these advancements remain unevenly distributed. While high-resource languages like English benefit from large-scale benchmarks, robust text classification for low-resource languages, such as Kazakh, continues to pose a significant challenge. This paper presents a comparative analysis of the classical Word2Vec embedding, BiLSTM recurrent models, BERT and XLM-R transformers, the generative mT5 model, alongside classical and statistical baselines for few-shot Kazakh news text classification under limited annotation constraints. Experiments were conducted on the KazNews dataset, curated for few-shot classification from tengrinews.kz and inform.kz articles, as well as parallel HuffPost corpora. The datasets comprise five categories (sports, politics, business, travel, etc.). The experimental findings in 1-shot and 5-shot settings demonstrate that no single model consistently dominates across all datasets and regimes: Word2Vec achieved the best performance on the HuffPost corpora in the 5-shot regime (0.48 and 0.53); mT5-Seq2Seq outperformed others on KazNews under 5-shot (0.71); while BERT remained competitive on the long-text KazNews dataset in the 5-shot setup. The authors also observed a sharp performance gain for the mT5-Seq2Seq model in the 5-shot setup. Conversely, none of the evaluated models yielded acceptable quality metrics in the 1-shot regime.

обработка естественного языканизкоресурсные языкиклассификация текстовfewshot learningрекуррентные моделитрансформерные моделигенеративные моделиfew-shot learningtext classificationKazakh languagelow-resource languagestransformer modelsBERTXLM-RBiLSTMWord2Vecdeep learningnatural language processingReferencesPakray, P., Gelbukh, A., & Bandyopadhyay, S. (2025). Natural language processing applications for low-resource languages. Natural Language Processing, 31(2), 183–197. https://doi.org/10.1017/nlp.2024.33Pakray, P., Gelbukh, A., & Bandyopadhyay, S. (2025). Natural language processing applications for low-resource languages. Natural Language Processing, 31(2), 183–197. https://doi.org/10.1017/nlp.2024.33Kowsari, K., Jafari Meimandi, K., Heidarysafa, M., Mendu, S., Barnes, L., & Brown, D. (2019). Text classification algorithms: A survey. Information, 10(4), 150. https://doi.org/10.3390/info10040150Kowsari, K., Jafari Meimandi, K., Heidarysafa, M., Mendu, S., Barnes, L., & Brown, D. (2019). Text classification algorithms: A survey. Information, 10(4), 150. https://doi.org/10.3390/info10040150McCallum, A., & Nigam, K. (1998). A comparison of event models for naive Bayes text classification. AAAI-98 Workshop on Learning for Text Categorization. https://aaai.org/papers/041-ws98-05-007/McCallum, A., & Nigam, K. (1998). A comparison of event models for naive Bayes text classification. AAAI-98 Workshop on Learning for Text Categorization. https://aaai.org/papers/041-ws98-05-007/Joachims, T. (1998). Text categorization with support vector machines: Learning with many relevant features. In C. Nédellec & C. Rouveirol (Eds.), Machine Learning: ECML-98 (pp. 137–142). Springer. https://doi.org/10.1007/BFb0026683Joachims, T. (1998). Text categorization with support vector machines: Learning with many relevant features. In C. Nédellec & C. Rouveirol (Eds.), Machine Learning: ECML-98 (pp. 137–142). Springer. https://doi.org/10.1007/BFb0026683Cox, D. R. (1958). The regression analysis of binary sequences. Journal of the Royal Statistical Society: Series B (Methodological), 20(2), 215–242.Cox, D. R. (1958). The regression analysis of binary sequences. Journal of the Royal Statistical Society: Series B (Methodological), 20(2), 215–242.Salton, G., Wong, A., & Yang, C.S. (1975). A vector space model for automatic indexing. Communications of the ACM, 18(11), 613–620. https://doi.org/10.1145/361219.361220Salton, G., Wong, A., & Yang, C.S. (1975). A vector space model for automatic indexing. Communications of the ACM, 18(11), 613–620. https://doi.org/10.1145/361219.361220Spärck Jones, K. (2004). A statistical interpretation of term specificity in retrieval. Journal of Documentation, 60(5), 493–502. https://doi.org/10.1108/00220410410560573Spärck Jones, K. (2004). A statistical interpretation of term specificity in retrieval. Journal of Documentation, 60(5), 493–502. https://doi.org/10.1108/00220410410560573Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural Computation, 9(8), 1735– 1780. https://doi.org/10.1162/neco.1997.9.8.1735Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural Computation, 9(8), 1735– 1780. https://doi.org/10.1162/neco.1997.9.8.1735Graves, A., & Schmidhuber, J. (2005). Framewise phoneme classification with bidirectional LSTM and other neural network architectures. Neural Networks, 18(5), 602–610. https://doi.org/10.1016/j.neunet.2005.06.042Graves, A., & Schmidhuber, J. (2005). Framewise phoneme classification with bidirectional LSTM and other neural network architectures. Neural Networks, 18(5), 602–610. https://doi.org/10.1016/j.neunet.2005.06.042Joulin, A., Grave, E., Bojanowski, P., & Mikolov, T. (2016). Bag of tricks for efficient text classification. arXiv. https://doi.org/10.48550/arXiv.1607.01759Joulin, A., Grave, E., Bojanowski, P., & Mikolov, T. (2016). Bag of tricks for efficient text classification. arXiv. https://doi.org/10.48550/arXiv.1607.01759Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013). Efficient estimation of word representations in vector space. arXiv. https://doi.org/10.48550/arXiv.1301.3781Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013). Efficient estimation of word representations in vector space. arXiv. https://doi.org/10.48550/arXiv.1301.3781Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Vol. 1, pp. 4171– 4186). Association for Computational Linguistics. https://doi.org/10.18653/v1/N19-1423Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Vol. 1, pp. 4171– 4186). Association for Computational Linguistics. https://doi.org/10.18653/v1/N19-1423Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F., ... & Zettlemoyer, L. (2019). Unsupervised cross-lingual representation learning at scale. arXiv. https://arxiv.org/abs/1911.02116Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F., ... & Zettlemoyer, L. (2019). Unsupervised cross-lingual representation learning at scale. arXiv. https://arxiv.org/abs/1911.02116Xue, L., Constant, N., Roberts, A., Kale, K., Al-Rfou, R., Siddhant, A., ... & Raffel, C. (2021). mT5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (pp. 483–498). https://doi.org/10.18653/v1/2021.naacl-main.41Xue, L., Constant, N., Roberts, A., Kale, K., Al-Rfou, R., Siddhant, A., ... & Raffel, C. (2021). mT5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (pp. 483–498). https://doi.org/10.18653/v1/2021.naacl-main.41Latief, A. D., Jarin, A., Yuyun, Hidayati, N. N., Afra, D. I. N., & Riza, H. (2025). A systematic review of few-shot and zero-shot learning for NLP in low-resource languages: Insights and challenges. IEEE Xplore. https://ieeexplore.ieee.org/abstract/document/11325582Latief, A. D., Jarin, A., Yuyun, Hidayati, N. N., Afra, D. I. N., & Riza, H. (2025). A systematic review of few-shot and zero-shot learning for NLP in low-resource languages: Insights and challenges. IEEE Xplore. https://ieeexplore.ieee.org/abstract/document/11325582Latief, A. D., Jarin, A., Yuyun, Hidayati, N. N., Afra, D. I. N., & Riza, H. (2025). A systematic review of few-shot and zero-shot learning for NLP in low-resource languages: Insights and challenges. In 2025 International Conference on Computer, Control, Informatics and its Applications (IC3INA) (pp. 358–363). IEEE. https://doi.org/10.1109/IC3INA68387.2025.11325582Latief, A. D., Jarin, A., Yuyun, Hidayati, N. N., Afra, D. I. N., & Riza, H. (2025). A systematic review of few-shot and zero-shot learning for NLP in low-resource languages: Insights and challenges. In 2025 International Conference on Computer, Control, Informatics and its Applications (IC3INA) (pp. 358–363). IEEE. https://doi.org/10.1109/IC3INA68387.2025.11325582Toleu, A., Tolegen, G., & Ualiyeva, I. (2025). Fine-Tuning Large Language Models for Kazakh Text Simplification. Applied Sciences, 15(15), 8344. https://doi.org/10.3390/app15158344Toleu, A., Tolegen, G., & Ualiyeva, I. (2025). Fine-Tuning Large Language Models for Kazakh Text Simplification. Applied Sciences, 15(15), 8344. https://doi.org/10.3390/app15158344

The authors declare that there are no conflicts of interest present.