OPTIMIZING SYNTACTIC-SEMANTIC RELATION EXTRACTION FOR THE KAZAKH LANGUAGE WITH TRANSFORMER ARCHITECTURES AND SYNTHETIC CORPORA

Natural language processing (NLP) methods are widely used in search engines, decision-support systems, and many other intelligent applications. One of the essential yet technically demanding tasks in this area is the extraction of triple relations in the form “subject–predicate–object.” Such structures are the basis for knowledge graphs and reasoning, but for languages with limited annotated resources, like Kazakh, this task becomes especially difficult. In our work, we investigate how the use of synthetic data can partially compensate for the lack of linguistic resources. The experimental setup included the generation of additional training data, followed by the training and testing of a model based on the Cross-lingual Language Model – Robustly Optimized BERT Approach (XLMRoBERTa) for triple extraction. XLM-RoBERTa, an improved version of the Bidirectional Encoder Representations from Transformers (BERT) model, benefits from a larger training corpus and increased size. This architecture is effective in cross-linguistic transfer tasks without additional fine-tuning, even between languages with different writing systems. The results show an F1-score of 90.73%. This indicates that even relatively simple augmentation strategies, when combined with advanced models, may considerably improve model performance when working with low-resource languages. The study also suggests that the approach can be extended to other underrepresented languages and integrated into practical systems for information retrieval and knowledge management.

1. Zhang, Y., Sadler, T., Taesiri, M.R., Xu, W., and Reformat, M. Fine-tuning language models for triple extraction with data augmentation. Proceedings of the 1st Workshop on Knowledge Graphs and Large Language Models (KaLLM 2024), 116–124 (2024).

2. Li, Z., Li, X., Sheng, J., and Slamu, W. AgglutiFiT: Efficient low-resource agglutinative language model fine-tuning. IEEE Access, 8, 148489–148499 (2020). https://doi.org/10.1109/ACCESS.2020.3015854

3. Barakhnin, V.B., Fedotov, A.M., Bakieva, A.M., Bakiev, M.N., Tazhibaeva, S.Zh., Batura, T.V., Kozhemyakina, O.Yu., Tusupov, D.A., Sambetbayeva, M.A., and Lukpanova, L.K. Algoritmy generatsii i stemmatizatsii slovoform kazakhskogo yazyka [Algorithms for generation and stemming of Kazakh language word forms]. Cloud of Science, 4 (2017). (in Russian).

4. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. BERT: Pre-training of deep bidirectional transformers for language understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 4171–4186 (2019). https://doi.org/10.18653/v1/N19-1423

5. Tolegen, G., Toleu, A., Mussabayev, R., Zhumazhanov, B., and Ziyatbekova, G. Generative pre-trained transformer for Kazakh text generation tasks. 2023 19th International Asian School-Seminar on Optimization Problems of Complex Systems (OPCS), 144–148 (2023). https://doi.org/10.1109/OPCS59592.2023.10275765

6. Rabchevsky, A.N., Ashikhmin, E.G., and Yasnitsky, L.N. Synthesis of datasets for neural networks based on expert knowledge. In: Arseniev, D.G., and Aouf, N. (eds.). Cyber-Physical Systems and Control II. Lecture Notes in Networks and Systems, 460, 561–569 (2023). https://doi.org/10.1007/978-3-031-20875-1_50

7. Karyukin, V., Rakhimova, D., Karibayeva, A., Turganbayeva, A., and Turarbek, A. The neural machine translation models for the low-resource Kazakh–English language pair. PeerJ Computer Science, 9, e1224 (2024). https://doi.org/10.7717/peerj-cs.1224

8. Akhmed-Zaki, D., Mansurova, M., Madiyeva, G., and Kyrgyzbayeva, M. Development of the information system for the Kazakh language preprocessing. Cogent Engineering, 8 (1), 1896418 (2021). https://doi.org/10.1080/23311916.2021.1896418

9. Abibullayeva, A., and Çetin, A. Keyword extraction from Kazakh news dataset with BERT. El-Cezeri Journal of Science and Engineering, 9 (4), 1193–1200 (2022). https://doi.org/10.31202/ecjse.1131826

10. Bektemyssova, G., and Sabdenov, A. Building a semantic knowledge graph search model for finding a causal answer. Revue d'Intelligence Artificielle, 38 (1), 243–250 (2024). https://doi.org/10.18280/ria.380125

11. Kaffee, L.-A., Biswas, R., Keet, C.M., Kalemi Vakaj, E., and de Melo, G. Multilingual knowledge graphs and low-resource languages: A review. Transactions on Graph Data and Knowledge, 1 (1), 10:1–10:19 (2024). https://doi.org/10.4230/TGDK.1.1.10

12. Chen, J., Geng, Y., Chen, Z., Pan, J., He, Y., Zhang, W., Horrocks, I., and Chen, H. Low-resource learning with knowledge graphs: A comprehensive survey. arXiv preprint, arXiv:2112.10006 (2021). https://doi.org/10.48550/arXiv.2112.10006

13. Kortmann, F., et al. Concept of a cloud state modeling system for lead-acid batteries: Theory and prototyping. 2021 International Conference on Electronics, Information, and Communication (ICEIC), 1–4 (2021). https://doi.org/10.1109/ICEIC51217.2021.9369785

14. Wang, Z., Karthikeyan, K., Mayhew, S., and Roth, D. Extending multilingual BERT to low-resource languages. Findings of the Association for Computational Linguistics: EMNLP 2020, 2649–2656 (2020). https://doi.org/10.18653/v1/2020.findings-emnlp.240

15. Conneau, A., Khandelwal, U., Goyal, N., Wenzek, G., Guzmán, F., Grave, E., Ott, M., Zettlemoyer, L., and Stoyanov, V. Unsupervised cross-lingual representation learning at scale. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 8440–8451 (2020). https://doi.org/10.18653/v1/2020.acl-main.747

16. Bhatt, A., Vaghela, N., and Dudhia, K. Generating knowledge graphs from large language models: A comparative study of GPT-4, LLaMA 2, and BERT. arXiv preprint, arXiv:2412.07412 (2024). https://doi.org/10.48550/arXiv.2412.07412

17. Deepchecks Community Blog. LLM models comparison: GPT-4, Bard, LLaMA, Flan-UL2, BLOOM (2024). URL: https://www.deepchecks.com/llm-models-comparison

18. OpenAI. GPT-4 technical report. arXiv preprint, arXiv:2303.08774 (2023). https://doi.org/10.48550/arXiv.2303.08774

19. Bojic, L., Kovacevic, P., and Cabarkapa, M. GPT-4 surpassing human performance in linguistic pragmatics. arXiv preprint, arXiv:2312.09545 (2023). https://doi.org/10.48550/arXiv.2312.09545

20. Baktash, J.A., and Dawodi, M. GPT-4: A review on advancements and opportunities in natural language processing. arXiv preprint, arXiv:2305.03195 (2023). https://doi.org/10.48550/arXiv.2305.03195

21. Guzev, V.G., and Burykin, A.A. Obshchie stroevye osobennosti agglutinativnykh yazykov [General structural features of agglutinative languages]. Acta Linguistica Petropolitana. Trudy Instituta Lingvisticheskikh Issledovaniy, 1 (2017). (in Russian).

22. Pires, T., Schlinger, E., and Garrette, D. How multilingual is multilingual BERT? Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 4996–5001 (2019).

23. Urmanov, N., Bektemyssova, G.U., and Basiri, K. NLP algorithms in OOP for processing the textbased documents in Russian language in machine learning. International Journal of Research, 6, 361–371 (2019).

24. Madasamy, A.K., Kumar, C.A., Dhanalakshmi, V., Rekha, R.U., Soman, K.P., and Rajendran, S. Morphological analyzer for agglutinative languages using machine learning approaches. 2009 International Conference on Advances in Recent Technologies in Communication and Computing (ARTCom), 433–435 (2009). https://doi.org/10.1109/ARTCom.2009.18

25. Ismayilzada, M., Circi, D., Sälevä, J., Sirin, H., Köksal, A., Dhingra, B., Bosselut, A., van der Plas, L., and Ataman, D. Evaluating morphological compositional generalization in large language models. arXiv preprint, arXiv:2410.12656 (2024). https://doi.org/10.48550/arXiv.2410.12656