DEVELOPMENT OF A MODEL FOR REAL-TIME RECOGNITION OF KAZAKH SIGN LANGUAGE USING MEDIAPIPE AND DEEP LEARNING METHODS

Technologies for automatic processing of sign language have become an urgent need for members of society with hearing and speech impairments who face inequality in the era of digital transformation. In recent years, the issue of considering sign language as a formal structure equal to natural language and adapting it to automatic systems has attracted increasing attention from researchers. To perform the task of automatically translating information from natural language into sign language, glosses, which are the textual representation of sign language, are used as an intermediate layer. For this purpose, this study proposes a new method for converting Kazakh language text, which reflects the morphological features of the Kazakh language, into sign language glosses using natural language processing techniques. In particular, a Seq2Seq architecture based on the ByT5 small model is applied. The obtained results demonstrate that the generated gloss sequences are compact and semantically rich while preserving the internal structure of sign language. The gloss sequence makes it possible to automate the work of an interpretable intermediate layer that represents sign language movements as logical units similar to written language. The transformed gloss sequence preserves the structure of sign language, reduces redundancy, and improves sentence coherence. Thus, the use of only semantically meaningful units to control sign language avatars reduces computational requirements. Short and semantically rich glosses serve as an effective resource for synthesizing hand movements in sign language.

1. FAQ. WFD – World Federation of the Deaf. URL: https://wfdeaf.org/contact/faqs/ (date of access: 20.09.2025).

2. Human rights. Deaf History Europe. URL: https://deafhistory.eu/index.php/component/zoo/item/human-rights#:~:text=,from%20deaf%20people%E2%80%99s%20human%20rights (date of access: 20.09.2025).

3. Six difficulties: Sign language avatar. DW Innovation. URL: https://innovation.dw.com/articles/sixdifficulties-sign-language-avatar (date of access: 20.09.2025).

4. Zou, Y., Lin, H. A basic General Service List for Chinese Sign Language. Journal of Deaf Studies and Deaf Education, 30(3), 405–418 (2025). https://doi.org/10.1093/jdsade/enaf012.

5. Caselli, N.K. et al. ASL-LEX: A lexical database of American Sign Language. Behavior research methods, 49(2), 784–801 (2017). https://doi.org/10.3758/s13428-016-0742-0.

6. Perlman, M. et al. Iconicity in signed and spoken vocabulary: a comparison between American Sign Language, British Sign Language, English, and Spanish. Frontiers in psychology, 9, 1433 (2018). https://doi.org/10.3389/fpsyg.2018.01433.

7. Zhao, Y. et al. Relationship between vocabulary knowledge and reading comprehension in deaf and hard of hearing students. The Journal of Deaf Studies and Deaf Education, 26(4), 546–555 (2021). https://doi.org/10.1093/deafed/enab023.

8. Convertino, C. et al. Word and world knowledge among deaf learners with and without cochlear implants. Journal of Deaf Studies and Deaf Education, 19(4), 471–483 (2014). https://doi.org/10.1093/deafed/enu024.

9. Hall, W.C. What you don’t know can hurt you: The risk of language deprivation by impairing sign language development in deaf children. Maternal and child health journal, 21(5), 961–965 (2017). https://doi.org/10.1007/s10995-017-2287-y.

10. Deaf community: Sign language equals rights. Human Rights Watch URL: https://www.hrw.org/news/2022/09/23/deaf-community-sign-language-equals-rights (date of access: 20.09.2025).

11. AI and machine translation: A threat to the deaf community. Deaf Journalism URL: https://www.deafjournalism.eu/ai-and-machine-translation-a-threat-to-the-deaf-community/ (date of access: 20.09.2025).

12. Duarte, A. et al. How2sign: a large-scale multimodal dataset for continuous american sign language. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2735–2744 (2021). https://doi.org/10.1109/CVPR46437.2021.00276 Dataset: http://how2sign.github.io/openaccess.thecvf. comhow2sign.github.io.

13. Li, D. et al. Word-level deep sign language recognition from video: A new large-scale dataset and methods comparison. Proceedings of the IEEE/CVF winter conference on applications of computer vision, 1459–1469 (2020). URL: https://openaccess.thecvf.com/content_WACV_2020/papers/Li_Word-level_Deep_Sign_Language_Recognition_from_Video_A_New_Large-scale_WACV_2020_paper.pdf. Dataset: https://dxli94.github.io/WLASL/openaccess.thecvf.comdxli94.github.io.

14. Athitsos, V. et al. The american sign language lexicon video dataset. 2008 IEEE computer society conference on computer vision and pattern recognition workshops. IEEE, 2008, pp. 1–8. Dataset: https://www.bu.edu/asllrp/av/dai-asllvd.html.

15. Albanie, S. et al. Bbc-oxford british sign language dataset. 2021. arXiv preprint arXiv:2111.03635 (2024). URL: https://www.robots.ox.ac.uk/~vgg/data/bobsl/(seearXiv:2111.03635).robots.ox.ac.ukarXiv.

16. Zhou, H. et al. Improving sign language translation with monolingual data by sign back-translation. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (2021), pp. 1316–1325. Dataset: CSL-Daily, https://ustc-slr.github.io/datasets/2021_csl_daily/openaccess.thecvf.comVisualSignLanguageResearch Group.

17. Ham, S. et al. Ksl-guide: A large-scale korean sign language dataset including interrogative sentences for guiding the deaf and hard-of-hearing. 2021 16th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2021). IEEE, 2021, pp. 1–8. Dataset: https://github.com/ChelseaGH/KSL-Guide (also listed at sign-lang@LREC). GitHubsign-lang.uni-hamburg.de.

18. Cormier, K., & Schembri, A. British Sign Language Corpus [Data collection]. UK Data Archive, 2018. https://doi.org/10.5255/UKDA-SN-851521reshare.ukdataservice.ac.uk.

19. Belissen, V., Braffort, A., Gouiffès, M. Dicta-Sign-LSF-v2: remake of a continuous French sign language dialogue corpus and a first baseline for automatic sign language processing. LREC 2020, 12th Conference on Language Resources and Evaluation (2020). URL: https://aclanthology.org/2020.lrec-1.740/. Dataset page: https://www.sign-lang.uni-hamburg.de/lr/compendium/corpus/dictasignlsfv2.htmlaclanthology.orgsign-lang.uni-hamburg.de.

20. Camgoz, N. C. et al. Neural sign language translation. Proceedings of the IEEE conference on computer vision and pattern recognition, 7784–7793 (2018). URL: https://www-i6.informatik.rwth-aachen.de/~koller/RWTH-PHOENIX-2014-T/www-i6.informatik.rwth-aachen.de.

21. Konrad, R., Hanke, T., Langer, G., Blanck, D., Bleicken, J., Hofmann, I., … Schulder, M. MEINE DGS – annotiert. Public Corpus of German Sign Language, 3rd release [Dataset]. Universität Hamburg, 2020. https://doi.org/10.25592/dgs.corpus-3.0sign-lang.uni-hamburg.de

22. Ye, J. et al. Scaling back-translation with domain text generation for sign language gloss translation. arXiv preprint arXiv:2210.07054 (2022) https://doi.org/10.18653/v1/2023.eacl-main.34.

23. Zhu, D., Czehmann, V., Avramidis, E. Neural machine translation methods for translating text to sign language glosses. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 12523–12541 (2023). https://doi.org/10.18653/v1/2023.acl-long.700.

24. Xu, C. et al. Automatic gloss dictionary for sign language learners. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, 83–92 (2022). https://doi.org/10.18653/v1/2022.acl-demo.8.

25. Mohamed, A. et al. A deep learning approach for gloss sign language translation using transformer. Journal of Computing and Communication, 1(2), 1–8 (2022).

26. Müller, M. et al. Considerations for meaningful sign language machine translation based on glosses. arXiv preprint arXiv:2211.15464 (2022).

27. Lin, K. et al. Gloss-free end-to-end sign language translation. arXiv preprint arXiv:2305.12876 (2023). https://doi.org/10.18653/v1/2023.acl-long.722.

28. Ye, J. et al. Cross-modality data augmentation for end-to-end sign language translation. arXiv preprint arXiv:2305.11096 (2023). https://doi.org/10.18653/v1/2023.findings-emnlp.904.

29. Shi, B. et al. Open-domain sign language translation learned from online video. arXiv preprint arXiv:2205.12870 (2022).

30. Kim, Y., Baek, H. Preprocessing for keypoint-based sign language translation without glosses. Sensors, 23(6), 3231 (2023).

31. Angelova, G., Avramidis, E., Möller, S. Using neural machine translation methods for sign language translation. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop, 273–284 (2022).

32. Cao, Y. et al. Explore more guidance: A task-aware instruction network for sign language translation enhanced with data augmentation. arXiv preprint arXiv:2204.05953 (2022).

33. Sousa, C., Coheur, L., Moita, M. Enhancing Accessible Communication: from European Portuguese to Portuguese Sign Language. Findings of the Association for Computational Linguistics: EMNLP 2023, 11452–11460 (2023).