kaz29Вестник Казахстанско-Британского технического университетаHerald of the Kazakh-British Technical University1998-66882959-8109Казахстанско-Британский Технический Университет10.55452/1998-6688-2025-22-4-10-22kaz29-2275Research ArticleКОМПЬЮТЕРНЫЕ НАУКИCOMPUTER SCIENCEСОЗДАНИЕ МОДЕЛИ ДЛЯ РАСПОЗНАВАНИЯ КАЗАХСКОГО ЖЕСТОВОГО ЯЗЫКА В РЕЖИМЕ РЕАЛЬНОГО ВРЕМЕНИ С ПОМОЩЬЮ MEDIAPIPE И МЕТОДОВ ГЛУБОКОГО ОБУЧЕНИЯDEVELOPMENT OF A MODEL FOR REAL-TIME RECOGNITION OF KAZAKH SIGN LANGUAGE USING MEDIAPIPE AND DEEP LEARNING METHODShttps://orcid.org/0000-0002-2013-1513ЕримбетоваА.YerimbetovaA.

к.т.н., ассоциированный профессор, PhD

г. Алматы

Cand. Tech. Sc., Associate Professor, PhD

Almaty

aigerian8888@gmail.comhttps://orcid.org/0009-0000-2467-5721БержановаУ. Г.BerzhanovaU. G.

докторант

г. Алматы

PhD student

Almaty

berzhanovaulmekenn@gmail.comhttps://orcid.org/0000-0002-4255-5456ДайырбаеваЭ.DaiyrbayevaE.

преподаватель

г. Алматы

Senior Lecturer

Almaty

nurbekkyzy_e@mail.ruhttps://orcid.org/0000-0002-9849-6176СакеновБ.SakenovB.

инженер-программист

г. Алматы

Software engineer

Almaty

sbakzhan22@gmail.comhttps://orcid.org/0000-0001-9358-1614СамбетбаеваМ.SambetbayevaM.

г. Астана

PhD

Astana

madina_jgtu@mail.ruИнститут информационных и вычислительных технологий КН МНВО РК; Казахский национальный университет им. аль-Фараби; Евразийский технологический университетКазахстанInstitute of Information and Computational Technologies of the Committee of Science of the Ministry of Science and Higher Education of the Republic of Kazakhstan; Al-Farabi Kazakh National University; Eurasian Technological UniversityKazakhstanКазахский национальный университет им. аль-ФарабиКазахстанAl-Farabi Kazakh National UniversityKazakhstanИнститут информационных и вычислительных технологий КН МНВО РК; Казахский национальный исследовательский технический университет им. К.И. СатпаеваКазахстанInstitute of Information and Computational Technologies of the Committee of Science of the Ministry of Science and Higher Education of the Republic of Kazakhstan; Satbayev UniversityKazakhstanИнститут информационных и вычислительных технологий КН МНВО РККазахстанInstitute of Information and Computational Technologies of the Committee of Science of the Ministry of Science and Higher Education of the Republic of KazakhstanKazakhstanЕвразийский университет им. Л.Н. ГумилеваКазахстанL.N. Gumilyov Eurasian National UniversityKazakhstan2025231220252241022Copyright © Еримбетова А., Бержанова У.Г., Дайырбаева Э., Сакенов Б., Самбетбаева М., 20252025Еримбетова А., Бержанова У.Г., Дайырбаева Э., Сакенов Б., Самбетбаева М.Yerimbetova A., Berzhanova U.G., Daiyrbayeva E., Sakenov B., Sambetbayeva M.Данная работа распространяется под лицензией Creative Commons Attribution 4.0.This work is licensed under a Creative Commons Attribution 4.0 License.https://vestnik.kbtu.edu.kz/jour/article/view/2275

В данной статье рассматривается процесс разработки системы распознавания казахского жестового языка с использованием платформы MediaPipe. Платформа позволяет эффективно распознавать жесты в реальном времени. Основное внимание уделено созданию моделей для распознавания жестов, обучению нейронных сетей и интеграции с платформой MediaPipe. Один из важных аспектов – достижение высокой точности и скорости обработки жестов с использованием архитектуры нейронных сетей. Система была обучена на большом наборе аннотированных жестов, что значительно улучшило качество распознавания. Для распознавания жестов казахского языка использовалась нейронная сеть типа LSTM, так как она эффективно работает с временными рядами и последовательностями данных. Модель была обучена на 30 жестах казахского жестового языка, что позволяет преобразовывать жесты в текст в реальном времени. Этот подход значительно облегчает коммуникацию с людьми, имеющими проблемы со слухом и речью, и способствует повышению инклюзивности. Кроме того, был разработан удобный веб-интерфейс, который позволяет легко интегрировать нейронную сеть с приложениями для распознавания жестов. Одним из ключевых аспектов работы является улучшение методов аннотирования и обработки данных для повышения точности распознавания. Будущее развитие системы включает расширение базы данных жестов и интеграцию с вебприложениями. Это позволит улучшить социальную интеграцию людей с нарушениями слуха и речи и создать широкую и доступную платформу.

This article discusses the process of developing a Kazakh sign language recognition system using the MediaPipe platform. The platform allows for efficient real-time gesture recognition. The main focus is on creating models for gesture recognition, training neural networks, and integrating with the MediaPipe platform. One of the key aspects is achieving high accuracy and speed in gesture processing by using neural network architecture. The system was trained on a large dataset of annotated gestures, which significantly improved the recognition quality. For recognizing Kazakh sign language gestures, an LSTM neural network was used because it effectively works with time series and data sequences. The model was trained on 30 Kazakh sign language gestures, enabling the conversion of gestures into text in real-time. This approach greatly facilitates communication with people who have hearing and speech impairments and contributes to increased inclusivity. Additionally, a user-friendly web interface was developed, allowing easy integration of the neural network with applications for gesture recognition. One of the key aspects of the work is improving data annotation and processing methods to enhance recognition accuracy. The future development of the system includes expanding the sign language gesture database and integration with web applications. This will improve social inclusion for people with hearing and speech impairments and create a broad, accessible platform.

казахский жестовый языкMediaPipeглубокое обучениеискусственная нейронная сетьрежим реального времениKazakh sign languageMediaPipedeep learningartificial neural networkin real-timeБұл зерттеу Қазақстан Республикасының Ғылым және жоғары білім министрлігінің Ғылым комитеті тарапынан қаржыландырылды (Грант № BR24992875).ReferencesSharma, S., Singh, S. Recognition of Indian sign language (ISL) using deep learning model. Wireless personal communications, 123 (1), 671–692 (2022). https://doi.org/10.1007/s11277-021-09152-1.Sharma, S., Singh, S. Recognition of Indian sign language (ISL) using deep learning model. Wireless personal communications, 123 (1), 671–692 (2022). https://doi.org/10.1007/s11277-021-09152-1.Mukhanov, S. et al. Gesture recognition of the Kazakh alphabet based on machine and deep learning models. Procedia Computer Science, 241, 458–463 (2024). https://doi.org/10.1016/j.procs.2024.08.064.Mukhanov, S. et al. Gesture recognition of the Kazakh alphabet based on machine and deep learning models. Procedia Computer Science, 241, 458–463 (2024). https://doi.org/10.1016/j.procs.2024.08.064.Wu, J. et al. Data glove-based gesture recognition using CNN-BiLSTM model with attention mechanism. Plos one, 18 (11), e0294174 (2023). https://doi.org/10.1371/journal.pone.0294174.Wu, J. et al. Data glove-based gesture recognition using CNN-BiLSTM model with attention mechanism. Plos one, 18 (11), e0294174 (2023). https://doi.org/10.1371/journal.pone.0294174.Mohamed, T. et al. Intelligent hand gesture recognition system empowered with CNN. 2022 International Conference on Cyber Resilience (ICCR). IEEE, 2022, pp. 1–8. https://doi.org/10.1109/iccr56254.2022.9995760.Mohamed, T. et al. Intelligent hand gesture recognition system empowered with CNN. 2022 International Conference on Cyber Resilience (ICCR). IEEE, 2022, pp. 1–8. https://doi.org/10.1109/iccr56254.2022.9995760.Mesbahi, S.C. et al. Hand gesture recognition based on various deep learning YOLO models. International Journal of Advanced Computer Science and Applications, 14 (40) (2023). https://doi.org/10.14569/ijacsa.2023.0140435.Mesbahi, S.C. et al. Hand gesture recognition based on various deep learning YOLO models. International Journal of Advanced Computer Science and Applications, 14 (40) (2023). https://doi.org/10.14569/ijacsa.2023.0140435.Doždor, Z. et al. TY-Net: Transforming YOLO for hand gesture recognition. IEEE access.2023, https:// doi.org/ 10.1109/access.2023.3341702.Doždor, Z. et al. TY-Net: Transforming YOLO for hand gesture recognition. IEEE access.2023, https:// doi.org/ 10.1109/access.2023.3341702.Zhang, Z., Wu, B., Jiang, Y. Gesture recognition system based on improved YOLO v3. 2022 7th International Conference on Intelligent Computing and Signal Processing (ICSP). IEEE, 2022, pp. 1540– 1543. https://doi.org/10.1109/icsp54964.2022.9778394.Zhang, Z., Wu, B., Jiang, Y. Gesture recognition system based on improved YOLO v3. 2022 7th International Conference on Intelligent Computing and Signal Processing (ICSP). IEEE, 2022, pp. 1540– 1543. https://doi.org/10.1109/icsp54964.2022.9778394.Ling, L., Tao, J., Wu, G. Research on gesture recognition based on YOLOv5. 2021 33rd Chinese Control and Decision Conference (CCDC). IEEE, 2021, pp. 801–806. https://doi.org/10.1109/ccdc52312.2021.9602731.Ling, L., Tao, J., Wu, G. Research on gesture recognition based on YOLOv5. 2021 33rd Chinese Control and Decision Conference (CCDC). IEEE, 2021, pp. 801–806. https://doi.org/10.1109/ccdc52312.2021.9602731.Kumar, R., Bajpai, A., Sinha, A. MediaPipe and cnns for real-time asl gesture recognition. arXiv preprint arXiv:2305.05296, 2023. https://doi.org/10.48550/arxiv.2305.05296.Kumar, R., Bajpai, A., Sinha, A. MediaPipe and cnns for real-time asl gesture recognition. arXiv preprint arXiv:2305.05296, 2023. https://doi.org/10.48550/arxiv.2305.05296.Chong, K.S., Subaramaniam, K., Al-Hadi, I. A.A.Q. Developing a Prototype Hand Gesture Recognition System in Interpreting American Sign. Interpretation, 3, 1 (2022). https://doi.org/10.5954/icarob.2024.gs7-4.Chong, K.S., Subaramaniam, K., Al-Hadi, I. A.A.Q. Developing a Prototype Hand Gesture Recognition System in Interpreting American Sign. Interpretation, 3, 1 (2022). https://doi.org/10.5954/icarob.2024.gs7-4.Grif, M.G., Kondratenko, Y.K. Recognition of Isolated Gestures of the Russian Sign Language Based on the Component Approach //2023 IEEE XVI International Scientific and Technical Conference Actual Problems of Electronic Instrument Engineering (APEIE). IEEE, 2023, pp. 1510–1513. https://doi.org/10.1109/apeie59731.2023.10347694.Grif, M.G., Kondratenko, Y.K. Recognition of Isolated Gestures of the Russian Sign Language Based on the Component Approach //2023 IEEE XVI International Scientific and Technical Conference Actual Problems of Electronic Instrument Engineering (APEIE). IEEE, 2023, pp. 1510–1513. https://doi.org/10.1109/apeie59731.2023.10347694.Samaan, G.H. et al. MediaPipe’s landmarks with rnn for dynamic sign language recognition. Electronics, 11 (19), 3228 (2022). https://doi.org/10.3390/electronics11193228.Samaan, G.H. et al. MediaPipe’s landmarks with rnn for dynamic sign language recognition. Electronics, 11 (19), 3228 (2022). https://doi.org/10.3390/electronics11193228.Zholshiyeva, L. et al. A Real-Time Approach to Recognition of Kazakh Sign Language //2022 International Conference on Smart Information Systems and Technologies (SIST). IEEE, 2022, pp. 1–6. https://doi.org/10.1109/sist54437.2022.9945799.Zholshiyeva, L. et al. A Real-Time Approach to Recognition of Kazakh Sign Language //2022 International Conference on Smart Information Systems and Technologies (SIST). IEEE, 2022, pp. 1–6. https://doi.org/10.1109/sist54437.2022.9945799.Amirgaliyev, Y., Ataniyazova, A., Buribayev, Z., Zhassuzak, M., Urmashev, B., & Cherikbayeva, L. Application of neural networks ensemble method for the Kazakh sign language recognition. Bulletin of Electrical Engineering and Informatics, 13(5), 3275–3287 (2024). https://doi.org/10.11591/eei.v13i5.7803.Amirgaliyev, Y., Ataniyazova, A., Buribayev, Z., Zhassuzak, M., Urmashev, B., & Cherikbayeva, L. Application of neural networks ensemble method for the Kazakh sign language recognition. Bulletin of Electrical Engineering and Informatics, 13(5), 3275–3287 (2024). https://doi.org/10.11591/eei.v13i5.7803.Yerimbetova, A., Sakenov, B., Berzhanova, U., Mukazhanov, N., Daiyrbayeva, E., & Othman, M. Development of A Model of Kazakh Sign Language Recognition Based on Deep Learning Method. In 2024 9th International Conference on Computer Science and Engineering (UBMK) (pp. 822-827). IEEE, 2024, October. https://doi.org/10.11591/eei.v13i5.780310.1109/UBMK63289.2024.10773578.Yerimbetova, A., Sakenov, B., Berzhanova, U., Mukazhanov, N., Daiyrbayeva, E., & Othman, M. Development of A Model of Kazakh Sign Language Recognition Based on Deep Learning Method. In 2024 9th International Conference on Computer Science and Engineering (UBMK) (pp. 822-827). IEEE, 2024, October. https://doi.org/10.11591/eei.v13i5.780310.1109/UBMK63289.2024.10773578.Yerimbetova, A., Sakenov, B., Sambetbayeva, M., Daiyrbayeva, E., Berzhanova, U., & Othman, M. Creating a Parallel Corpus for the Kazakh Sign Language and Learning. Applied Sciences, 15(5), 2808 (2025). https://doi.org/10.3390/app15052808.Yerimbetova, A., Sakenov, B., Sambetbayeva, M., Daiyrbayeva, E., Berzhanova, U., & Othman, M. Creating a Parallel Corpus for the Kazakh Sign Language and Learning. Applied Sciences, 15(5), 2808 (2025). https://doi.org/10.3390/app15052808.Vijitkunsawat, W., Anunvrapong, P., & Chantngarm, P. Human Joint Coordinate Sequencing in VideoBased Thai Finger Spelling Recognition. In 2024 8th International Conference on Information Technology (InCIT) (pp. 681–686). IEEE, 2024, November. https://doi.org/10.1109/InCIT63192.2024.10810526.Vijitkunsawat, W., Anunvrapong, P., & Chantngarm, P. Human Joint Coordinate Sequencing in VideoBased Thai Finger Spelling Recognition. In 2024 8th International Conference on Information Technology (InCIT) (pp. 681–686). IEEE, 2024, November. https://doi.org/10.1109/InCIT63192.2024.10810526.Katti, R.K., Sujatha, C., Desai, P., & Shankar, G. Character and word level gesture recognition of Indian Sign language. In 2023 IEEE 8th International Conference for Convergence in Technology (I2CT) (pp. 1–6). IEEE, 2023, April. https://doi.org/10.1109/I2CT57861.2023.10126314.Katti, R.K., Sujatha, C., Desai, P., & Shankar, G. Character and word level gesture recognition of Indian Sign language. In 2023 IEEE 8th International Conference for Convergence in Technology (I2CT) (pp. 1–6). IEEE, 2023, April. https://doi.org/10.1109/I2CT57861.2023.10126314.Feng, Y., Chen, N., Wu, Y., Jiang, C., Liu, S., & Chen, S. DFCNet+: Cross-modal dynamic feature contrast net for continuous sign language recognition. Image and Vision Computing, 151, 105260 (2024). https://doi.org/10.1016/j.imavis.2024.105260.Feng, Y., Chen, N., Wu, Y., Jiang, C., Liu, S., & Chen, S. DFCNet+: Cross-modal dynamic feature contrast net for continuous sign language recognition. Image and Vision Computing, 151, 105260 (2024). https://doi.org/10.1016/j.imavis.2024.105260.Aleshin, N.A. Recurrent neural networks. World science: problems and innovations, pp. 10–12, 2021.Aleshin, N.A. Recurrent neural networks. World science: problems and innovations, pp. 10–12, 2021.

The authors declare that there are no conflicts of interest present.