kaz29Вестник Казахстанско-Британского технического университетаHerald of the Kazakh-British Technical University1998-66882959-8109Казахстанско-Британский Технический Университет10.55452/1998-6688-2025-22-4-155-167kaz29-2291Research ArticleКОМПЬЮТЕРНЫЕ НАУКИCOMPUTER SCIENCEФОРМАЛИЗАЦИЯ ТЕКСТА НА КАЗАХСКОМ ЯЗЫКЕ С ИСПОЛЬЗОВАНИЕМ ГЛОССАРНОГО СЛОЯDEVELOPMENT OF A MODEL FOR REAL-TIME RECOGNITION OF KAZAKH SIGN LANGUAGE USING MEDIAPIPE AND DEEP LEARNING METHODShttps://orcid.org/0000-0002-4669-9254АмангелдіН.AmangeldyN.

PhD

г. Алматы

г. Астана

PhD

Almaty

Astana

amangeldi_n_3@enu.kzhttps://orcid.org/0000-0002-2013-1513ЕримбетоваА.YerimbetovaA.

 к.т.н., ассоциированный профессор, PhD

г. Алматы

Cand. Tech. Sc., PhD, Associate Professor

Almaty

 

aigerian8888@gmail.comhttps://orcid.org/0009-0007-7278-4202ГазизоваН. Е.GazizovaN.

магистр

г. Алматы

г. Астана

MSc

Almaty

Astana

g.nazerke2755@gmail.comhttps://orcid.org/0000-0002-1857-9028ТурсыноваН. А.TursynovaN.

магистр

г. Алматы

г. Астана

MSc

Almaty

Astana

tursynova_na@enu.kzhttps://orcid.org/0009-0000-7659-4974БолатбекқызыК.BolatbekkyzyK.

бакалавр

г. Астана

Bachelor

Astana

kerbezbolatbekkyzy03@gmail.comИнститут информационных и вычислительных технологий; Евразийский национальный университет им. Л.Н. Гумилева; ТОО «SignBridge»КазахстанInstitute of Information and Computational Technologies; L.N. Gumilyov Eurasian National University; «SignBridge» LLPKazakhstanИнститут информационных и вычислительных технологий; Евразийский технологический университетКазахстанInstitute of Information and Computational Technologies; Eurasian Technological UniversityKazakhstanИнститут информационных и вычислительных технологий; ТОО «SignBridge»КазахстанInstitute of Information and Computational Technologies; «SignBridge» LLPKazakhstanИнститут информационных и вычислительных технологий; Евразийский национальный университет им. Л.Н. ГумилеваКазахстанInstitute of Information and Computational Technologies; L.N. Gumilyov Eurasian National UniversityKazakhstanТОО «SignBridge»Казахстан«SignBridge» LLPKazakhstan202523122025224155167Copyright © Амангелді Н., Еримбетова А., Газизова Н.Е., Турсынова Н.А., Болатбекқызы К., 20252025Амангелді Н., Еримбетова А., Газизова Н.Е., Турсынова Н.А., Болатбекқызы К.Amangeldy N., Yerimbetova A., Gazizova N., Tursynova N., Bolatbekkyzy K.Данная работа распространяется под лицензией Creative Commons Attribution 4.0.This work is licensed under a Creative Commons Attribution 4.0 License.https://vestnik.kbtu.edu.kz/jour/article/view/2291

Технологии автоматической обработки жестового языка стали актуальной потребностью членов общества с нарушениями слуха и речи, которые сталкиваются с неравенством в эпоху цифровой трансформации. В последние годы вопрос рассмотрения жестового языка как формальной структуры, равной естественному языку, и его адаптации к автоматическим системам привлекает повышенное внимание исследователей. Для выполнения задачи автоматического перевода информации с естественного языка на жестовый язык в качестве промежуточного слоя используются глоссы, то есть текстовая форма жестового языка. В данном исследовании предлагается новый метод преобразования текста на казахском языке, учитывающий морфологические особенности казахского языка, в глоссы жестового языка с использованием методов обработки естественного языка. В частности, применяется архитектура Seq2Seq на основе модели ByT5 small. Полученные результаты показывают, что сформированные последовательности глосс являются компактными и семантически насыщенными, сохраняя внутреннюю структуру жестового языка. Последовательность глосс позволяет автоматизировать работу интерпретируемого промежуточного слоя, представляющего жестовые движения как логические единицы, аналогичные письменному языку. Преобразованная последовательность глосс сохраняет структуру жестового языка, уменьшает избыточность и повышает связность предложения. Таким образом, использование только семантически значимых единиц при управлении аватарами жестового языка снижает вычислительные затраты. Короткие и семантически насыщенные глоссы являются эффективным ресурсом для синтеза движений рук в жестовом языке.

Technologies for automatic processing of sign language have become an urgent need for members of society with hearing and speech impairments who face inequality in the era of digital transformation. In recent years, the issue of considering sign language as a formal structure equal to natural language and adapting it to automatic systems has attracted increasing attention from researchers. To perform the task of automatically translating information from natural language into sign language, glosses, which are the textual representation of sign language, are used as an intermediate layer. For this purpose, this study proposes a new method for converting Kazakh language text, which reflects the morphological features of the Kazakh language, into sign language glosses using natural language processing techniques. In particular, a Seq2Seq architecture based on the ByT5 small model is applied. The obtained results demonstrate that the generated gloss sequences are compact and semantically rich while preserving the internal structure of sign language. The gloss sequence makes it possible to automate the work of an interpretable intermediate layer that represents sign language movements as logical units similar to written language. The transformed gloss sequence preserves the structure of sign language, reduces redundancy, and improves sentence coherence. Thus, the use of only semantically meaningful units to control sign language avatars reduces computational requirements. Short and semantically rich glosses serve as an effective resource for synthesizing hand movements in sign language.

жестовый языкпоследовательность глоссобработка естественного языкаформализацияSeq2SeqByT5sign languagegloss sequencenatural language processingformalizationSeq2SeqByT5Бұл зерттеу Қазақстан Республикасының Ғылым және жоғары білім министрлігінің Ғылым комитеті тарапынан қаржыландырылды (Грант № BR24992875).ReferencesFAQ. WFD – World Federation of the Deaf. URL: https://wfdeaf.org/contact/faqs/ (date of access: 20.09.2025).FAQ. WFD – World Federation of the Deaf. URL: https://wfdeaf.org/contact/faqs/ (date of access: 20.09.2025).Human rights. Deaf History Europe. URL: https://deafhistory.eu/index.php/component/zoo/item/human-rights#:~:text=,from%20deaf%20people%E2%80%99s%20human%20rights (date of access: 20.09.2025).Human rights. Deaf History Europe. URL: https://deafhistory.eu/index.php/component/zoo/item/human-rights#:~:text=,from%20deaf%20people%E2%80%99s%20human%20rights (date of access: 20.09.2025).Six difficulties: Sign language avatar. DW Innovation. URL: https://innovation.dw.com/articles/sixdifficulties-sign-language-avatar (date of access: 20.09.2025).Six difficulties: Sign language avatar. DW Innovation. URL: https://innovation.dw.com/articles/sixdifficulties-sign-language-avatar (date of access: 20.09.2025).Zou, Y., Lin, H. A basic General Service List for Chinese Sign Language. Journal of Deaf Studies and Deaf Education, 30(3), 405–418 (2025). https://doi.org/10.1093/jdsade/enaf012.Zou, Y., Lin, H. A basic General Service List for Chinese Sign Language. Journal of Deaf Studies and Deaf Education, 30(3), 405–418 (2025). https://doi.org/10.1093/jdsade/enaf012.Caselli, N.K. et al. ASL-LEX: A lexical database of American Sign Language. Behavior research methods, 49(2), 784–801 (2017). https://doi.org/10.3758/s13428-016-0742-0.Caselli, N.K. et al. ASL-LEX: A lexical database of American Sign Language. Behavior research methods, 49(2), 784–801 (2017). https://doi.org/10.3758/s13428-016-0742-0.Perlman, M. et al. Iconicity in signed and spoken vocabulary: a comparison between American Sign Language, British Sign Language, English, and Spanish. Frontiers in psychology, 9, 1433 (2018). https://doi.org/10.3389/fpsyg.2018.01433.Perlman, M. et al. Iconicity in signed and spoken vocabulary: a comparison between American Sign Language, British Sign Language, English, and Spanish. Frontiers in psychology, 9, 1433 (2018). https://doi.org/10.3389/fpsyg.2018.01433.Zhao, Y. et al. Relationship between vocabulary knowledge and reading comprehension in deaf and hard of hearing students. The Journal of Deaf Studies and Deaf Education, 26(4), 546–555 (2021). https://doi.org/10.1093/deafed/enab023.Zhao, Y. et al. Relationship between vocabulary knowledge and reading comprehension in deaf and hard of hearing students. The Journal of Deaf Studies and Deaf Education, 26(4), 546–555 (2021). https://doi.org/10.1093/deafed/enab023.Convertino, C. et al. Word and world knowledge among deaf learners with and without cochlear implants. Journal of Deaf Studies and Deaf Education, 19(4), 471–483 (2014). https://doi.org/10.1093/deafed/enu024.Convertino, C. et al. Word and world knowledge among deaf learners with and without cochlear implants. Journal of Deaf Studies and Deaf Education, 19(4), 471–483 (2014). https://doi.org/10.1093/deafed/enu024.Hall, W.C. What you don’t know can hurt you: The risk of language deprivation by impairing sign language development in deaf children. Maternal and child health journal, 21(5), 961–965 (2017). https://doi.org/10.1007/s10995-017-2287-y.Hall, W.C. What you don’t know can hurt you: The risk of language deprivation by impairing sign language development in deaf children. Maternal and child health journal, 21(5), 961–965 (2017). https://doi.org/10.1007/s10995-017-2287-y.Deaf community: Sign language equals rights. Human Rights Watch URL: https://www.hrw.org/news/2022/09/23/deaf-community-sign-language-equals-rights (date of access: 20.09.2025).Deaf community: Sign language equals rights. Human Rights Watch URL: https://www.hrw.org/news/2022/09/23/deaf-community-sign-language-equals-rights (date of access: 20.09.2025).AI and machine translation: A threat to the deaf community. Deaf Journalism URL: https://www.deafjournalism.eu/ai-and-machine-translation-a-threat-to-the-deaf-community/ (date of access: 20.09.2025).AI and machine translation: A threat to the deaf community. Deaf Journalism URL: https://www.deafjournalism.eu/ai-and-machine-translation-a-threat-to-the-deaf-community/ (date of access: 20.09.2025).Duarte, A. et al. How2sign: a large-scale multimodal dataset for continuous american sign language. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2735–2744 (2021). https://doi.org/10.1109/CVPR46437.2021.00276 Dataset: http://how2sign.github.io/openaccess.thecvf. comhow2sign.github.io.Duarte, A. et al. How2sign: a large-scale multimodal dataset for continuous american sign language. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2735–2744 (2021). https://doi.org/10.1109/CVPR46437.2021.00276 Dataset: http://how2sign.github.io/openaccess.thecvf. comhow2sign.github.io.Li, D. et al. Word-level deep sign language recognition from video: A new large-scale dataset and methods comparison. Proceedings of the IEEE/CVF winter conference on applications of computer vision, 1459–1469 (2020). URL: https://openaccess.thecvf.com/content_WACV_2020/papers/Li_Word-level_Deep_Sign_Language_Recognition_from_Video_A_New_Large-scale_WACV_2020_paper.pdf. Dataset: https://dxli94.github.io/WLASL/openaccess.thecvf.comdxli94.github.io.Li, D. et al. Word-level deep sign language recognition from video: A new large-scale dataset and methods comparison. Proceedings of the IEEE/CVF winter conference on applications of computer vision, 1459–1469 (2020). URL: https://openaccess.thecvf.com/content_WACV_2020/papers/Li_Word-level_Deep_Sign_Language_Recognition_from_Video_A_New_Large-scale_WACV_2020_paper.pdf. Dataset: https://dxli94.github.io/WLASL/openaccess.thecvf.comdxli94.github.io.Athitsos, V. et al. The american sign language lexicon video dataset. 2008 IEEE computer society conference on computer vision and pattern recognition workshops. IEEE, 2008, pp. 1–8. Dataset: https://www.bu.edu/asllrp/av/dai-asllvd.html.Athitsos, V. et al. The american sign language lexicon video dataset. 2008 IEEE computer society conference on computer vision and pattern recognition workshops. IEEE, 2008, pp. 1–8. Dataset: https://www.bu.edu/asllrp/av/dai-asllvd.html.Albanie, S. et al. Bbc-oxford british sign language dataset. 2021. arXiv preprint arXiv:2111.03635 (2024). URL: https://www.robots.ox.ac.uk/~vgg/data/bobsl/(seearXiv:2111.03635).robots.ox.ac.ukarXiv.Albanie, S. et al. Bbc-oxford british sign language dataset. 2021. arXiv preprint arXiv:2111.03635 (2024). URL: https://www.robots.ox.ac.uk/~vgg/data/bobsl/(seearXiv:2111.03635).robots.ox.ac.ukarXiv.Zhou, H. et al. Improving sign language translation with monolingual data by sign back-translation. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (2021), pp. 1316–1325. Dataset: CSL-Daily, https://ustc-slr.github.io/datasets/2021_csl_daily/openaccess.thecvf.comVisualSignLanguageResearch Group.Zhou, H. et al. Improving sign language translation with monolingual data by sign back-translation. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (2021), pp. 1316–1325. Dataset: CSL-Daily, https://ustc-slr.github.io/datasets/2021_csl_daily/openaccess.thecvf.comVisualSignLanguageResearch Group.Ham, S. et al. Ksl-guide: A large-scale korean sign language dataset including interrogative sentences for guiding the deaf and hard-of-hearing. 2021 16th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2021). IEEE, 2021, pp. 1–8. Dataset: https://github.com/ChelseaGH/KSL-Guide (also listed at sign-lang@LREC). GitHubsign-lang.uni-hamburg.de.Ham, S. et al. Ksl-guide: A large-scale korean sign language dataset including interrogative sentences for guiding the deaf and hard-of-hearing. 2021 16th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2021). IEEE, 2021, pp. 1–8. Dataset: https://github.com/ChelseaGH/KSL-Guide (also listed at sign-lang@LREC). GitHubsign-lang.uni-hamburg.de.Cormier, K., & Schembri, A. British Sign Language Corpus [Data collection]. UK Data Archive, 2018. https://doi.org/10.5255/UKDA-SN-851521reshare.ukdataservice.ac.uk.Cormier, K., & Schembri, A. British Sign Language Corpus [Data collection]. UK Data Archive, 2018. https://doi.org/10.5255/UKDA-SN-851521reshare.ukdataservice.ac.uk.Belissen, V., Braffort, A., Gouiffès, M. Dicta-Sign-LSF-v2: remake of a continuous French sign language dialogue corpus and a first baseline for automatic sign language processing. LREC 2020, 12th Conference on Language Resources and Evaluation (2020). URL: https://aclanthology.org/2020.lrec-1.740/. Dataset page: https://www.sign-lang.uni-hamburg.de/lr/compendium/corpus/dictasignlsfv2.htmlaclanthology.orgsign-lang.uni-hamburg.de.Belissen, V., Braffort, A., Gouiffès, M. Dicta-Sign-LSF-v2: remake of a continuous French sign language dialogue corpus and a first baseline for automatic sign language processing. LREC 2020, 12th Conference on Language Resources and Evaluation (2020). URL: https://aclanthology.org/2020.lrec-1.740/. Dataset page: https://www.sign-lang.uni-hamburg.de/lr/compendium/corpus/dictasignlsfv2.htmlaclanthology.orgsign-lang.uni-hamburg.de.Camgoz, N. C. et al. Neural sign language translation. Proceedings of the IEEE conference on computer vision and pattern recognition, 7784–7793 (2018). URL: https://www-i6.informatik.rwth-aachen.de/~koller/RWTH-PHOENIX-2014-T/www-i6.informatik.rwth-aachen.de.Camgoz, N. C. et al. Neural sign language translation. Proceedings of the IEEE conference on computer vision and pattern recognition, 7784–7793 (2018). URL: https://www-i6.informatik.rwth-aachen.de/~koller/RWTH-PHOENIX-2014-T/www-i6.informatik.rwth-aachen.de.Konrad, R., Hanke, T., Langer, G., Blanck, D., Bleicken, J., Hofmann, I., … Schulder, M. MEINE DGS – annotiert. Public Corpus of German Sign Language, 3rd release [Dataset]. Universität Hamburg, 2020. https://doi.org/10.25592/dgs.corpus-3.0sign-lang.uni-hamburg.deKonrad, R., Hanke, T., Langer, G., Blanck, D., Bleicken, J., Hofmann, I., … Schulder, M. MEINE DGS – annotiert. Public Corpus of German Sign Language, 3rd release [Dataset]. Universität Hamburg, 2020. https://doi.org/10.25592/dgs.corpus-3.0sign-lang.uni-hamburg.deYe, J. et al. Scaling back-translation with domain text generation for sign language gloss translation. arXiv preprint arXiv:2210.07054 (2022) https://doi.org/10.18653/v1/2023.eacl-main.34.Ye, J. et al. Scaling back-translation with domain text generation for sign language gloss translation. arXiv preprint arXiv:2210.07054 (2022) https://doi.org/10.18653/v1/2023.eacl-main.34.Zhu, D., Czehmann, V., Avramidis, E. Neural machine translation methods for translating text to sign language glosses. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 12523–12541 (2023). https://doi.org/10.18653/v1/2023.acl-long.700.Zhu, D., Czehmann, V., Avramidis, E. Neural machine translation methods for translating text to sign language glosses. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 12523–12541 (2023). https://doi.org/10.18653/v1/2023.acl-long.700.Xu, C. et al. Automatic gloss dictionary for sign language learners. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, 83–92 (2022). https://doi.org/10.18653/v1/2022.acl-demo.8.Xu, C. et al. Automatic gloss dictionary for sign language learners. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, 83–92 (2022). https://doi.org/10.18653/v1/2022.acl-demo.8.Mohamed, A. et al. A deep learning approach for gloss sign language translation using transformer. Journal of Computing and Communication, 1(2), 1–8 (2022).Mohamed, A. et al. A deep learning approach for gloss sign language translation using transformer. Journal of Computing and Communication, 1(2), 1–8 (2022).Müller, M. et al. Considerations for meaningful sign language machine translation based on glosses. arXiv preprint arXiv:2211.15464 (2022).Müller, M. et al. Considerations for meaningful sign language machine translation based on glosses. arXiv preprint arXiv:2211.15464 (2022).Lin, K. et al. Gloss-free end-to-end sign language translation. arXiv preprint arXiv:2305.12876 (2023). https://doi.org/10.18653/v1/2023.acl-long.722.Lin, K. et al. Gloss-free end-to-end sign language translation. arXiv preprint arXiv:2305.12876 (2023). https://doi.org/10.18653/v1/2023.acl-long.722.Ye, J. et al. Cross-modality data augmentation for end-to-end sign language translation. arXiv preprint arXiv:2305.11096 (2023). https://doi.org/10.18653/v1/2023.findings-emnlp.904.Ye, J. et al. Cross-modality data augmentation for end-to-end sign language translation. arXiv preprint arXiv:2305.11096 (2023). https://doi.org/10.18653/v1/2023.findings-emnlp.904.Shi, B. et al. Open-domain sign language translation learned from online video. arXiv preprint arXiv:2205.12870 (2022).Shi, B. et al. Open-domain sign language translation learned from online video. arXiv preprint arXiv:2205.12870 (2022).Kim, Y., Baek, H. Preprocessing for keypoint-based sign language translation without glosses. Sensors, 23(6), 3231 (2023).Kim, Y., Baek, H. Preprocessing for keypoint-based sign language translation without glosses. Sensors, 23(6), 3231 (2023).Angelova, G., Avramidis, E., Möller, S. Using neural machine translation methods for sign language translation. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop, 273–284 (2022).Angelova, G., Avramidis, E., Möller, S. Using neural machine translation methods for sign language translation. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop, 273–284 (2022).Cao, Y. et al. Explore more guidance: A task-aware instruction network for sign language translation enhanced with data augmentation. arXiv preprint arXiv:2204.05953 (2022).Cao, Y. et al. Explore more guidance: A task-aware instruction network for sign language translation enhanced with data augmentation. arXiv preprint arXiv:2204.05953 (2022).Sousa, C., Coheur, L., Moita, M. Enhancing Accessible Communication: from European Portuguese to Portuguese Sign Language. Findings of the Association for Computational Linguistics: EMNLP 2023, 11452–11460 (2023).Sousa, C., Coheur, L., Moita, M. Enhancing Accessible Communication: from European Portuguese to Portuguese Sign Language. Findings of the Association for Computational Linguistics: EMNLP 2023, 11452–11460 (2023).

The authors declare that there are no conflicts of interest present.