DEVELOPMENT OF A MODEL FOR REAL-TIME RECOGNITION OF KAZAKH SIGN LANGUAGE USING MEDIAPIPE AND DEEP LEARNING METHODS

This article discusses the process of developing a Kazakh sign language recognition system using the MediaPipe platform. The platform allows for efficient real-time gesture recognition. The main focus is on creating models for gesture recognition, training neural networks, and integrating with the MediaPipe platform. One of the key aspects is achieving high accuracy and speed in gesture processing by using neural network architecture. The system was trained on a large dataset of annotated gestures, which significantly improved the recognition quality. For recognizing Kazakh sign language gestures, an LSTM neural network was used because it effectively works with time series and data sequences. The model was trained on 30 Kazakh sign language gestures, enabling the conversion of gestures into text in real-time. This approach greatly facilitates communication with people who have hearing and speech impairments and contributes to increased inclusivity. Additionally, a user-friendly web interface was developed, allowing easy integration of the neural network with applications for gesture recognition. One of the key aspects of the work is improving data annotation and processing methods to enhance recognition accuracy. The future development of the system includes expanding the sign language gesture database and integration with web applications. This will improve social inclusion for people with hearing and speech impairments and create a broad, accessible platform.

1. Sharma, S., Singh, S. Recognition of Indian sign language (ISL) using deep learning model. Wireless personal communications, 123 (1), 671–692 (2022). https://doi.org/10.1007/s11277-021-09152-1.

2. Mukhanov, S. et al. Gesture recognition of the Kazakh alphabet based on machine and deep learning models. Procedia Computer Science, 241, 458–463 (2024). https://doi.org/10.1016/j.procs.2024.08.064.

3. Wu, J. et al. Data glove-based gesture recognition using CNN-BiLSTM model with attention mechanism. Plos one, 18 (11), e0294174 (2023). https://doi.org/10.1371/journal.pone.0294174.

4. Mohamed, T. et al. Intelligent hand gesture recognition system empowered with CNN. 2022 International Conference on Cyber Resilience (ICCR). IEEE, 2022, pp. 1–8. https://doi.org/10.1109/iccr56254.2022.9995760.

5. Mesbahi, S.C. et al. Hand gesture recognition based on various deep learning YOLO models. International Journal of Advanced Computer Science and Applications, 14 (40) (2023). https://doi.org/10.14569/ijacsa.2023.0140435.

6. Doždor, Z. et al. TY-Net: Transforming YOLO for hand gesture recognition. IEEE access.2023, https:// doi.org/ 10.1109/access.2023.3341702.

7. Zhang, Z., Wu, B., Jiang, Y. Gesture recognition system based on improved YOLO v3. 2022 7th International Conference on Intelligent Computing and Signal Processing (ICSP). IEEE, 2022, pp. 1540– 1543. https://doi.org/10.1109/icsp54964.2022.9778394.

8. Ling, L., Tao, J., Wu, G. Research on gesture recognition based on YOLOv5. 2021 33rd Chinese Control and Decision Conference (CCDC). IEEE, 2021, pp. 801–806. https://doi.org/10.1109/ccdc52312.2021.9602731.

9. Kumar, R., Bajpai, A., Sinha, A. MediaPipe and cnns for real-time asl gesture recognition. arXiv preprint arXiv:2305.05296, 2023. https://doi.org/10.48550/arxiv.2305.05296.

10. Chong, K.S., Subaramaniam, K., Al-Hadi, I. A.A.Q. Developing a Prototype Hand Gesture Recognition System in Interpreting American Sign. Interpretation, 3, 1 (2022). https://doi.org/10.5954/icarob.2024.gs7-4.

11. Grif, M.G., Kondratenko, Y.K. Recognition of Isolated Gestures of the Russian Sign Language Based on the Component Approach //2023 IEEE XVI International Scientific and Technical Conference Actual Problems of Electronic Instrument Engineering (APEIE). IEEE, 2023, pp. 1510–1513. https://doi.org/10.1109/apeie59731.2023.10347694.

12. Samaan, G.H. et al. MediaPipe’s landmarks with rnn for dynamic sign language recognition. Electronics, 11 (19), 3228 (2022). https://doi.org/10.3390/electronics11193228.

13. Zholshiyeva, L. et al. A Real-Time Approach to Recognition of Kazakh Sign Language //2022 International Conference on Smart Information Systems and Technologies (SIST). IEEE, 2022, pp. 1–6. https://doi.org/10.1109/sist54437.2022.9945799.

14. Amirgaliyev, Y., Ataniyazova, A., Buribayev, Z., Zhassuzak, M., Urmashev, B., & Cherikbayeva, L. Application of neural networks ensemble method for the Kazakh sign language recognition. Bulletin of Electrical Engineering and Informatics, 13(5), 3275–3287 (2024). https://doi.org/10.11591/eei.v13i5.7803.

15. Yerimbetova, A., Sakenov, B., Berzhanova, U., Mukazhanov, N., Daiyrbayeva, E., & Othman, M. Development of A Model of Kazakh Sign Language Recognition Based on Deep Learning Method. In 2024 9th International Conference on Computer Science and Engineering (UBMK) (pp. 822-827). IEEE, 2024, October. https://doi.org/10.11591/eei.v13i5.780310.1109/UBMK63289.2024.10773578.

16. Yerimbetova, A., Sakenov, B., Sambetbayeva, M., Daiyrbayeva, E., Berzhanova, U., & Othman, M. Creating a Parallel Corpus for the Kazakh Sign Language and Learning. Applied Sciences, 15(5), 2808 (2025). https://doi.org/10.3390/app15052808.

17. Vijitkunsawat, W., Anunvrapong, P., & Chantngarm, P. Human Joint Coordinate Sequencing in VideoBased Thai Finger Spelling Recognition. In 2024 8th International Conference on Information Technology (InCIT) (pp. 681–686). IEEE, 2024, November. https://doi.org/10.1109/InCIT63192.2024.10810526.

18. Katti, R.K., Sujatha, C., Desai, P., & Shankar, G. Character and word level gesture recognition of Indian Sign language. In 2023 IEEE 8th International Conference for Convergence in Technology (I2CT) (pp. 1–6). IEEE, 2023, April. https://doi.org/10.1109/I2CT57861.2023.10126314.

19. Feng, Y., Chen, N., Wu, Y., Jiang, C., Liu, S., & Chen, S. DFCNet+: Cross-modal dynamic feature contrast net for continuous sign language recognition. Image and Vision Computing, 151, 105260 (2024). https://doi.org/10.1016/j.imavis.2024.105260.

20. Aleshin, N.A. Recurrent neural networks. World science: problems and innovations, pp. 10–12, 2021.