A LANGUAGE MODEL INTEGRATING ONTOLOGICAL AND CORPUS-DATA

This study addresses the urgent need to analyze digital threats in everyday discourse by constructing a 1,000text annotated corpus from social media and news platforms covering military and geopolitical events. The purpose of the proposed study is to address the urgent need to analyze digital threats in everyday discourse by creating an annotated corpus of texts with elements of information operations. A multi-layered annotation scheme captures semantic actors and pragmatic features – including impact type, emotional tone, disinformation markers, and intent (e.g., provocation, intimidation). Annotation via Label Studio ensured flexibility, quality control, and context sensitivity, with inter-annotator reliability (Cohen’s Kappa = 0.82) confirming consistency. In pilot experiments, the Onto-IO-BERT model achieved an F1-score of 0.81, outperforming baseline classifiers. Practical utility was validated through analysis of real Telegram messages. The framework is tailored for studying military information operations within Kazakhstan’s Ministry of Internal Affairs and the created corps is a new resource for analyzing military information operations, filling a significant gap in the existing data set. The presented corpus contains texts in Kazakh, Russian and English. The corpus is openly accessible at: https://github.com/baiangali/multi_mil

1. Yerimbetova, A., Sakenov, B., Sambetbayeva, M., Daiyrbayeva, E., Berzhanova, U., & Othman, M. Creating a Parallel Corpus for the Kazakh Sign Language and Learning. Applied Sciences, 15(5), 2808 (2025). https://doi.org/10.3390/app15052808

2. Sadirmekova, Zh., Daiyrbayeva, E., Karbozova, I., & Bekbolatov, S. Development of a method for automatic extraction of ontology entity names from natural language texts. In UBMK 2024 Proceedings: 9th International Conference on Computer Science and Engineering, IEEE, 74–79 (2024). https://doi.org/10.1109/UBMK63289.2024.10773503

3. Mussiraliyeva, S., Bolatbek, M., Zhumakhanova, A., Medetbek, Z., & Sagynay, M. Comparative analysis of machine learning algorithms to identify extremist texts in the Kazakh language. Scientific Journal of Astana IT University, 14(14), 71–90 (2023). https://doi.org/10.37943/14DKRN4681

4. Mussiraliyeva, S., Bolatbek, M., Zhumakhanova, A., Baispay, G., & Medetbek, Z. Creating a model of semantic analysis of extremist texts in the Kazakh language. Journal of Mathematics, Mechanics and Computer Science, 121(1), 110–121 (2024). https://doi.org/10.26577/JMMCS2024121111

5. Rzayeva, L., Pogolovkin, D., & Myrzatay, A. Development of a modular messaging analysis service based on NLP for digital forensics. News of the NAS RK. Series of Physics and Mathematics, 2, 212–233 (2025). https://doi.org/10.32014/2025.2518-1726.354 (in Russian).

6. Modzelewski, A., Da San Martino, G., Savov, P., Wilczyńska, M. A., & Wierzbicki, A. MIPD: Exploring manipulation and intention in a novel corpus of disinformation. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 1234–1245 (2024). ACL.

7. Klein, I. Information operations and influence campaigns. IEEE Phoenix Computer Society, February 2024.

8. Sambetbayeva, M., Nekessova, A., Yerimbetova, A., Bayangali, A., Kaldarova, M., Telman, D., & Smailov, N. A Multi-Level Annotation Model for Fake News Detection: Implementing Kazakh-Russian Corpus via Label Studio. Big Data and Cognitive Computing, 9(8), 215 (2025). https://doi.org/10.3390/bdcc9080215

9. Lin, Y., Wang, H., & Celikyilmaz, A. A survey on recent advances in named entity recognition from deep learning models. ACM Computing Surveys, 55(2), 1–40 (2023). https://doi.org/10.1145/3514221

10. Derczynski, R., Bontcheva, L., & Roberts, K. Broad Twitter corpus: A diverse named entity recognition dataset. In Proceedings of COLING 2016, 1169–1179 (2016).

11. Barrón-Cedeño, A., Rosso, P., & Yu, S. Fine-grained analysis of propaganda in news articles. In Proceedings of EMNLP 2019 , 564–573 (2019).

12. Rashkin, H., Choi, E., Jang, J. Y., Volkova, S., & Choi, Y. Truth of varying shades: Analyzing language in fake news and political fact-checking. In Proceedings of EMNLP 2017, 2921–2927 (2017).

13. Horn, C., Wiegand, M., & Klakow, D. Towards a multidimensional model of media bias in news articles. In Proceedings of COLING 2018, 498–509 (2018).

14. Reisigl, M., & Wodak, R. The discourse-historical approach. In Methods of Critical Discourse Studies, 3rd ed., 23–61 (2015).

15. Van Dijk, T. Critical discourse studies: A sociocognitive approach. In Methods of Critical Discourse Analysis , 62–86 (2011).

16. Zeldes, A. The GUM corpus: Creating multilayer resources in a university setting. Language Resources and Evaluation, 51, 581–612 (2017). https://doi.org/10.1007/s10579-016-9343-4

17. Al-Rawi, A. Framing the Syrian conflict on Twitter. Global Media and Communication, 10 (2), 153– 170 (2014). https://doi.org/10.1177/1742766514536022

18. Vego, E. Effects-based operations: A critique. Joint Forces Quarterly, 41, 51–57 (2006).

19. Dandeker, C. Military language and strategic doctrine. Sociological Review, 54(3), 581–598 (2006). https://doi.org/10.1111/j.1467-954X.2006.00628.x

20. Röttger, P., Schröder, M., Grotov, A., & Augenstein, I. Harmful but legal: Cognitive framing and large language models in online information warfare. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024). ACL.

21. Walker, C., Strassel, S., Medero, J., & Maeda, K. ACE 2005 multilingual training corpus. Linguistic Data Consortium. https://catalog.ldc.upenn.edu/LDC2006T06

22. Wang, X., Wang, Z., Han, X., Jiang, W., Han, R., Liu, Z., Li, J., Li, P., Lin, Y., & Zhou, J. MAVEN: A massive general domain event detection dataset. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP),1652–1671 (2020).

23. Li, X., Li, F., Pan, L., Chen, Y., Peng, W., Wang, Q., Lyu, Y., & Zhu, Y. DuEE: A large-scale dataset for Chinese event extraction in real-world scenarios. In X. Zhu, M. Zhang, Y. Hong, & R. He (Eds.), Natural Language Processing and Chinese Computing, 534–545 (2020).

24. Ebner, S., Xia, P., Culkin, R., Rawlins, K., & Van Durme, B. Multi-sentence argument linking. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 8057–8077 (2020).

25. Li, S., Ji, H., & Han, J. Document-level event argument extraction by conditional generation. In Proceedings of NAACL-HLT 2021, 894–908 (2021).

26. Tong, M., Xu, B., Wang, S., Han, M., Cao, Y., Zhu, J., Chen, S., Hou, L., & Li, J. DocEE: A largescale and fine-grained benchmark for document-level event extraction. In Proceedings of NAACL-HLT 2022, 3970–3982 (2022).

27. Huang, H., Sun, J., Wei, H., Xiao, K., Wang, M., & Li, X. A dataset of domain events based on opensource military news. China Scientific Data, 8(1), 30 (2023). https://doi.org/10.11922/sciencedata.2023.0005.zh

28. Zhu, M., Xu, Z., Zeng, K., Xiao, K., Wang, M., Ke, W., & Huang, H. CMNEE: A large-scale documentlevel event extraction dataset based on open-source Chinese military news. In Proceedings of LREC-COLING 2024, 3367–3379 (2024).

29. Chen, L.-C., Chang, K.-H., & Yang, S.-C. Integrating corpus-based and NLP approach to extract terminology and domain-oriented information: An example of US military corpus. Acta Scientiarum. Technology, 44, 2022. https://doi.org/10.4025/actascitechnol.v44i1.60486

30. Li, H., Zhu, S.-C., & Zheng, Z. (2023). DiPlomat: A dialogue dataset for situated pragmatic reasoning. arXiv preprint. https://arxiv.org/abs/2306.09030

31. Label Studio Documentation. Label Studio annotator guide, 2023. https://labelstud.io/guide

32. International Organization for Standardization. ISO 24617-1:2012 – Language resource management – Semantic annotation framework (SemAF) – Part 1: Time and events (ISO-TimeML). https://www.iso.org/standard/37331.html

33. International Organization for Standardization. ISO 24617-6:2016 – Language resource management – Semantic annotation framework – Part 6: Principles of semantic annotation (SemAF Principles). https://www.iso.org/standard/60581.html

34. Wardle, C., & Derakhshan, H. Information disorder: Toward an interdisciplinary framework for research and policymaking. Council of Europe, 2017. https://shorensteincenter.org/information-disorderframework-for-research-and-policymaking/

35. Joint Chiefs of Staff. (2012). Joint publication 3-13: Information operations. U.S. Department of Defense. https://irp.fas.org/doddir/dod/jp3_13.pdf

36. President of the Republic of Kazakhstan. (2023, March 20). On the approval of the information doctrine of the Republic of Kazakhstan (Decree No. 145). https://adilet.zan.kz/rus/docs/U2300000145

37. Schmitt, M. N. (Ed.). (2017). Tallinn Manual 2.0 on the international law applicable to cyber operations. Cambridge University Press.

38. Artstein, R., & Poesio, M. Survey article: Inter-coder agreement for computational linguistics. Computational Linguistics, 34(4), 555–596 (2008). https://doi.org/10.1162/coli.07-034-R2