kaz29Вестник Казахстанско-Британского технического университетаHerald of the Kazakh-British Technical University1998-66882959-8109Казахстанско-Британский Технический Университет10.55452/1998-6688-2026-23-2-187-204kaz29-2897Research ArticleКОМПЬЮТЕРНЫЕ НАУКИCOMPUTER SCIENCEМЕТОДЫ УПРАВЛЕНИЯ ИНТЕЛЛЕКТУАЛЬНЫМ ПРОИЗВОДСТВОМ С ИСПОЛЬЗОВАНИЕМ ОБУЧЕНИЯ С ПОДКРЕПЛЕНИЕМREINFORCEMENT LEARNING–DRIVEN CONTROL STRATEGIES FOR SMART MANUFACTURING SYSTEMShttps://orcid.org/0000-0002-5862-6415СамигулинаЗ. И.SamigulinaZ. I.

PhD.

Алматы

PhD.

Almaty

z.samigulina@kbtu.kzhttps://orcid.org/0009-0001-2788-6521ДюсенкуловаБ. Ж.DyussenkulovaB. Z.

PhD докторант.

Алматы

PhD student.

Almaty

bu_dyussenkulova@kbtu.kzhttps://orcid.org/0009-0006-8151-4928БутаковаД. А.ButakovaD. A.

Магистрант.

Алматы

Master’s student.

Almaty

d.butakova@kbtu.kzКазахстанско-Британский технический университетКазахстанKazakh-British Technical UniversityKazakhstan202627062026232187204Copyright © Самигулина З.И., Дюсенкулова Б.Ж., Бутакова Д.А., 20262026Самигулина З.И., Дюсенкулова Б.Ж., Бутакова Д.А.Samigulina Z.I., Dyussenkulova B.Z., Butakova D.A.Данная работа распространяется под лицензией Creative Commons Attribution 4.0.This work is licensed under a Creative Commons Attribution 4.0 License.https://vestnik.kbtu.edu.kz/jour/article/view/2897

В настоящее время создание умных производственных систем является актуальной задачей. Широкое распространение получили нейронные сети, которые позволяют решать сложные производственные проблемы. Статья посвящена исследованию нейронных сетей с подкреплением DQN, PPO для диагностики состояния промышленного оборудования в рамках модели GEMMA. Французский подход GEMMA построен на основе языка SFC (Sequential Function Charts) и содержит стандарты по управлению технологическими процессами. Предлагается внедрение нейронных сетей в зону Д, модели GEMMA. Результаты моделирования и экспериментов осуществлялись на основе двух баз данных, одна сгенерирована искусственным путем, вторая взята с промышленного производства. Применение рассмотренных архитектур позволяет добиться хороших результатов для работы с индустриальными данными.

Nowadays, the creation of smart manufacturing systems has high importance. Neural networks have been widely applied to solve complex manufacturing challenges. The paper is devoted to the study of neural networks with reinforcement learning as PPO (Proximal Policy Optimization), DQN (Deep Q-network) for state diagnosis of industrial equipment within the GEMMA (Guide d’Etude des Modes de Marche er d’Arret) model. The GEMMA French approach is established on the SFC (Sequential Function Charts) language and includes standards for controlling technical processes. An application of neural networks in area D of the GEMMA model is introduced. Modelling and experimental results were conducted based on synthetic and experimental datasets. The implementation of the architecture considered allows us to achieve reliable results for industrial data.

умная производственная системаобучение с подкреплениемproximal policy optimizationdeep Q-networkGuide d’Etude des Modes de Marche er d’Arret (GEMMA) модельSmart manufacturing systemreinforcement learningproximal policy optimization (PPO)deep Q-network (DQN)informer transformerGuide d’Etude des Modes de Marche er d’Arret (GEMMA) modeldiagnostic of industrial equipmentReferencesJayakumar, S., and Nandakumar, S. Distributed resource optimization using the Q-learning algorithm in device-to-device communication: A reinforcement learning paradigm. Results in Engineering, 23, 102462 (2024). https://doi.org/10.1016/j.rineng.2024.102462Jayakumar, S., and Nandakumar, S. Distributed resource optimization using the Q-learning algorithm in device-to-device communication: A reinforcement learning paradigm. Results in Engineering, 23, 102462 (2024). https://doi.org/10.1016/j.rineng.2024.102462Xia, K., Sacco, C., Kirkpatrick, M., et al. A digital twin to train deep reinforcement learning agent for smart manufacturing plants: Environment, interfaces and intelligence. Journal of Manufacturing Systems, 58, 210–230 (2021). https://doi.org/10.1016/j.jmsy.2020.06.012Xia, K., Sacco, C., Kirkpatrick, M., et al. A digital twin to train deep reinforcement learning agent for smart manufacturing plants: Environment, interfaces and intelligence. Journal of Manufacturing Systems, 58, 210–230 (2021). https://doi.org/10.1016/j.jmsy.2020.06.012Park, K.T., Son, Y.H., Ko, S.W., and Noh, S.D. Digital twin and reinforcement learning-based resilient production control for micro smart factory. Applied Sciences, 11 (7) (2021). https://doi.org/10.3390/app11072977Park, K.T., Son, Y.H., Ko, S.W., and Noh, S.D. Digital twin and reinforcement learning-based resilient production control for micro smart factory. Applied Sciences, 11 (7) (2021). https://doi.org/10.3390/app11072977Ullah, I., Hassan, U.U., and Ali, M.I. Multi-level federated learning for Industry 4.0: A crowdsourcing approach. Procedia Computer Science, 217, 423–435 (2022). https://doi.org/10.1016/j.procs.2022.12.238Ullah, I., Hassan, U.U., and Ali, M.I. Multi-level federated learning for Industry 4.0: A crowdsourcing approach. Procedia Computer Science, 217, 423–435 (2022). https://doi.org/10.1016/j.procs.2022.12.238Mezair, T., Djenouri, Y., and Belhadi, A. A sustainable deep learning framework for fault detection in 6G Industry 4.0 heterogeneous data environments. Computer Communications, 187, 164–171 (2022). https://doi.org/10.1016/j.comcom.2022.02.010Mezair, T., Djenouri, Y., and Belhadi, A. A sustainable deep learning framework for fault detection in 6G Industry 4.0 heterogeneous data environments. Computer Communications, 187, 164–171 (2022). https://doi.org/10.1016/j.comcom.2022.02.010Neupane, D., and Seok, J. Bearing fault detection and diagnosis using Case Western Reserve University dataset with deep learning approaches: A review. IEEE Access, 8, 93155–93178 (2020). https://doi.org/10.1109/ACCESS.2020.2990528Neupane, D., and Seok, J. Bearing fault detection and diagnosis using Case Western Reserve University dataset with deep learning approaches: A review. IEEE Access, 8, 93155–93178 (2020). https://doi.org/10.1109/ACCESS.2020.2990528Mattera, G., Mattera, R., Vespoli, S., and Salatiello, E. Anomaly detection in manufacturing systems with temporal networks and unsupervised machine learning. Computers & Industrial Engineering, 203, 111023 (2025). https://doi.org/10.1016/j.cie.2025.111023Mattera, G., Mattera, R., Vespoli, S., and Salatiello, E. Anomaly detection in manufacturing systems with temporal networks and unsupervised machine learning. Computers & Industrial Engineering, 203, 111023 (2025). https://doi.org/10.1016/j.cie.2025.111023Prabu, S., Senthilraja, R., Ali, A.M., Jayapoorani, S., and Arun, M. AI-driven predictive maintenance for smart manufacturing systems using digital twin technology. International Journal of Computational and Experimental Science and Engineering, 11 (1), 1350–1355 (2025). https://doi.org/10.22399/ijcesen.1099Prabu, S., Senthilraja, R., Ali, A.M., Jayapoorani, S., and Arun, M. AI-driven predictive maintenance for smart manufacturing systems using digital twin technology. International Journal of Computational and Experimental Science and Engineering, 11 (1), 1350–1355 (2025). https://doi.org/10.22399/ijcesen.1099Asmat, H., Din, I., Almogren, A., and Khan, M. Digital twin with soft actor-critic reinforcement learning for transitioning from Industry 4.0 to 5.0. IEEE Access (2025). https://doi.org/10.1109/ACCESS.2025.3546085Asmat, H., Din, I., Almogren, A., and Khan, M. Digital twin with soft actor-critic reinforcement learning for transitioning from Industry 4.0 to 5.0. IEEE Access (2025). https://doi.org/10.1109/ACCESS.2025.3546085Ruiz Rodríguez, M.L., Kubler, S., de Giorgio, A., Cordy, M., Robert, J., and Le Traon, Y. Multi-agent deep reinforcement learning based predictive maintenance on parallel machines. Robotics and Computer-Integrated Manufacturing, 78, 102406 (2022). https://doi.org/10.1016/j.rcim.2022.102406Ruiz Rodríguez, M.L., Kubler, S., de Giorgio, A., Cordy, M., Robert, J., and Le Traon, Y. Multi-agent deep reinforcement learning based predictive maintenance on parallel machines. Robotics and Computer-Integrated Manufacturing, 78, 102406 (2022). https://doi.org/10.1016/j.rcim.2022.102406Pliego Marugán, A., Pinar-Pérez, J.M., and García Márquez, F.P. A reinforcement learning agent for maintenance of deteriorating systems with increasingly imperfect repairs. Reliability Engineering & System Safety, 252, 110466 (2024). https://doi.org/10.1016/j.ress.2024.110466Pliego Marugán, A., Pinar-Pérez, J.M., and García Márquez, F.P. A reinforcement learning agent for maintenance of deteriorating systems with increasingly imperfect repairs. Reliability Engineering & System Safety, 252, 110466 (2024). https://doi.org/10.1016/j.ress.2024.110466Serrano-Ruiz, J.C., Mula, J., and Poler, R. Development of a multidimensional conceptual model for job shop smart manufacturing scheduling from the Industry 4.0 perspective. Journal of Manufacturing Systems, 63, 185–202 (2022). https://doi.org/10.1016/j.jmsy.2022.03.011Serrano-Ruiz, J.C., Mula, J., and Poler, R. Development of a multidimensional conceptual model for job shop smart manufacturing scheduling from the Industry 4.0 perspective. Journal of Manufacturing Systems, 63, 185–202 (2022). https://doi.org/10.1016/j.jmsy.2022.03.011Zhou, L., Zhang, L., and Horn, B. Deep reinforcement learning-based dynamic scheduling in smart manufacturing. Procedia CIRP, 93, 383–388 (2020). https://doi.org/10.1016/j.procir.2020.05.163Zhou, L., Zhang, L., and Horn, B. Deep reinforcement learning-based dynamic scheduling in smart manufacturing. Procedia CIRP, 93, 383–388 (2020). https://doi.org/10.1016/j.procir.2020.05.163Zhang, M., Lu, Y., Hu, Y., et al. Dynamic scheduling method for job-shop manufacturing systems by deep reinforcement learning with proximal policy optimization. Sustainability, 14 (9) (2022). https://doi.org/10.3390/su14095177Zhang, M., Lu, Y., Hu, Y., et al. Dynamic scheduling method for job-shop manufacturing systems by deep reinforcement learning with proximal policy optimization. Sustainability, 14 (9) (2022). https://doi.org/10.3390/su14095177Yang, S., and Xu, Z. Intelligent scheduling and reconfiguration via deep reinforcement learning in smart manufacturing. International Journal of Production Research, 60 (16), 4936–4953 (2022). https://doi.org/10.1080/00207543.2021.1943037Yang, S., and Xu, Z. Intelligent scheduling and reconfiguration via deep reinforcement learning in smart manufacturing. International Journal of Production Research, 60 (16), 4936–4953 (2022). https://doi.org/10.1080/00207543.2021.1943037Ho, G., Tang, Y., Leung, E., and Tong, P. Integrated reinforcement learning of automated guided vehicles dynamic path planning for smart logistics and operations. Transportation Research Part E: Logistics and Transportation Review, 196 (2025). https://doi.org/10.1016/j.tre.2025.104008Ho, G., Tang, Y., Leung, E., and Tong, P. Integrated reinforcement learning of automated guided vehicles dynamic path planning for smart logistics and operations. Transportation Research Part E: Logistics and Transportation Review, 196 (2025). https://doi.org/10.1016/j.tre.2025.104008Zhou, T., Tang, D., and Wang, L. Reinforcement learning with composite rewards for production scheduling in a smart factory. IEEEAccess, 9, 752–766 (2021). https://doi.org/10.1109/ACCESS.2020.3046784Zhou, T., Tang, D., and Wang, L. Reinforcement learning with composite rewards for production scheduling in a smart factory. IEEEAccess, 9, 752–766 (2021). https://doi.org/10.1109/ACCESS.2020.3046784Bin Shaikat, F., and Ahmed Faysal, S. Optimization of production scheduling in smart manufacturing environments using machine learning algorithms (2025).Bin Shaikat, F., and Ahmed Faysal, S. Optimization of production scheduling in smart manufacturing environments using machine learning algorithms (2025).Kim, Y., Kim, H., Ha, B., and Kim, W. Federated digital twins: A scheduling approach based on temporal graph neural network and deep reinforcement learning. IEEE Access (2025). https://doi.org/10.1109/ACCESS.2025.3530558Kim, Y., Kim, H., Ha, B., and Kim, W. Federated digital twins: A scheduling approach based on temporal graph neural network and deep reinforcement learning. IEEE Access (2025). https://doi.org/10.1109/ACCESS.2025.3530558Chen, Y., Zhang, J., and Rauf, M. Dynamic scheduling of hybrid flow shop problem with uncertain process time and flexible maintenance using NeuroEvolution of Augmenting Topologies. IET Collaborative Intelligent Manufacturing, 6 (3) (2024). https://doi.org/10.1049/cim2.12119Chen, Y., Zhang, J., and Rauf, M. Dynamic scheduling of hybrid flow shop problem with uncertain process time and flexible maintenance using NeuroEvolution of Augmenting Topologies. IET Collaborative Intelligent Manufacturing, 6 (3) (2024). https://doi.org/10.1049/cim2.12119Schoepp, S., Taghian, M., and Miwa, S. Enhancing hardware fault tolerance in machines with reinforcement learning policy gradient algorithms (2024). https://doi.org/10.48550/arXiv.2407.15283Schoepp, S., Taghian, M., and Miwa, S. Enhancing hardware fault tolerance in machines with reinforcement learning policy gradient algorithms (2024). https://doi.org/10.48550/arXiv.2407.15283Modirrousta, M.H., Aliyari Shoorehdeli, M., and Yari, M. Imbalanced classification in faulty turbine data: New proximal policy optimisation. IET Collaborative Intelligent Manufacturing, 6 (3) (2024). https://doi.org/10.1049/cim2.12114Modirrousta, M.H., Aliyari Shoorehdeli, M., and Yari, M. Imbalanced classification in faulty turbine data: New proximal policy optimisation. IET Collaborative Intelligent Manufacturing, 6 (3) (2024). https://doi.org/10.1049/cim2.12114Yang, Y., Juntao, L., and Lingling, P. Multi-robot path planning based on a deep reinforcement learning DQN algorithm. CAAI Transactions on Intelligence Technology, 5 (3), 177–183 (2020). https://doi.org/10.1049/trit.2020.0024Yang, Y., Juntao, L., and Lingling, P. Multi-robot path planning based on a deep reinforcement learning DQN algorithm. CAAI Transactions on Intelligence Technology, 5 (3), 177–183 (2020). https://doi.org/10.1049/trit.2020.0024Zhu, Q., Han, J., Chai, K., and Zhao, C. Time series analysis based on Informer algorithms: A survey. Symmetry, 15 (4) (2023). https://doi.org/10.3390/sym15040951Zhu, Q., Han, J., Chai, K., and Zhao, C. Time series analysis based on Informer algorithms: A survey. Symmetry, 15 (4) (2023). https://doi.org/10.3390/sym15040951Zhang, X., Yang, K., and Zheng, L. Transformer fault diagnosis method based on TimesNet and Informer. Actuators, 13 (2) (2024). https://doi.org/10.3390/act13020074Zhang, X., Yang, K., and Zheng, L. Transformer fault diagnosis method based on TimesNet and Informer. Actuators, 13 (2) (2024). https://doi.org/10.3390/act13020074Pan, Y., et al. Understanding and mitigating the limitations of prioritized experience replay (2023). https://doi.org/10.48550/arXiv.2007.09569Pan, Y., et al. Understanding and mitigating the limitations of prioritized experience replay (2023). https://doi.org/10.48550/arXiv.2007.09569Bu, F., and Chang, D.E. Double prioritized state recycled experience replay. In: Proceedings of the IEEE International Conference on Consumer Electronics Asia (2020). https://doi.org/10.1109/ICCEAsia49877.2020.9276975Bu, F., and Chang, D.E. Double prioritized state recycled experience replay. In: Proceedings of the IEEE International Conference on Consumer Electronics Asia (2020). https://doi.org/10.1109/ICCEAsia49877.2020.9276975Chandrinos, N., Loi, I., Zachos, P., et al. Effectiveness of L2 regularization in privacy-preserving machine learning (2024). https://doi.org/10.48550/arXiv.2412.01541Chandrinos, N., Loi, I., Zachos, P., et al. Effectiveness of L2 regularization in privacy-preserving machine learning (2024). https://doi.org/10.48550/arXiv.2412.01541Alinda Rahmi, N., and Defit, S. The use of hyperparameter tuning in model classification: A scientific work area identification. International Journal of Applied Sciences and Smart Technologies, 8 (4) (2025). http://dx.doi.org/10.62527/joiv.8.4.3092Alinda Rahmi, N., and Defit, S. The use of hyperparameter tuning in model classification: A scientific work area identification. International Journal of Applied Sciences and Smart Technologies, 8 (4) (2025). http://dx.doi.org/10.62527/joiv.8.4.3092Shekhar, S., Bansode, A., and Salim, A. A comparative study of hyper-parameter optimization tools. In: 2021 IEEE Asia-Pacific Conference on Computer Science and Data Engineering (CSDE) (2021). https://doi.org/10.1109/CSDE53843.2021.9718485Shekhar, S., Bansode, A., and Salim, A. A comparative study of hyper-parameter optimization tools. In: 2021 IEEE Asia-Pacific Conference on Computer Science and Data Engineering (CSDE) (2021). https://doi.org/10.1109/CSDE53843.2021.9718485Jordon, J., Szpruch, L., Houssiau, F., et al. Synthetic data – what, why and how? (2022). https://doi.org/10.48550/arXiv.2205.03257Jordon, J., Szpruch, L., Houssiau, F., et al. Synthetic data – what, why and how? (2022). https://doi.org/10.48550/arXiv.2205.03257Fan, C., Chen, M., Wang, X., et al. A review on data preprocessing techniques toward efficient and reliable knowledge discovery from building operational data. Frontiers in Energy Research, 9 (2021). https://doi.org/10.3389/fenrg.2021.652801Fan, C., Chen, M., Wang, X., et al. A review on data preprocessing techniques toward efficient and reliable knowledge discovery from building operational data. Frontiers in Energy Research, 9 (2021). https://doi.org/10.3389/fenrg.2021.652801Bekar, E.T., Nyqvist, P., and Skoogh, A. An intelligent approach for data pre-processing and analysis in predictive maintenance with an industrial case study. Advances in Mechanical Engineering, 12 (5) (2020). https://doi.org/10.1177/168781402091Bekar, E.T., Nyqvist, P., and Skoogh, A. An intelligent approach for data pre-processing and analysis in predictive maintenance with an industrial case study. Advances in Mechanical Engineering, 12 (5) (2020). https://doi.org/10.1177/168781402091Ou, F., Wang, H., Zhang, C., et al. Industrial data-driven machine learning soft sensing for optimal operation of etching tools. Digital Chemical Engineering, 13 (2024). https://doi.org/10.1016/j.dche.2024.100195Ou, F., Wang, H., Zhang, C., et al. Industrial data-driven machine learning soft sensing for optimal operation of etching tools. Digital Chemical Engineering, 13 (2024). https://doi.org/10.1016/j.dche.2024.100195Terven, J., Cordova-Esparza, D., Ramirez-Pedraza, A., et al. Loss functions and metrics in deep learning (2023). https://doi.org/10.48550/arXiv.2307.02694Terven, J., Cordova-Esparza, D., Ramirez-Pedraza, A., et al. Loss functions and metrics in deep learning (2023). https://doi.org/10.48550/arXiv.2307.02694Le, P.B., and Nguyen, Z.T. ROC curves, loss functions, and distorted probabilities in binary classification. Mathematics, 10 (9) (2022). https://doi.org/10.3390/math10091410Le, P.B., and Nguyen, Z.T. ROC curves, loss functions, and distorted probabilities in binary classification. Mathematics, 10 (9) (2022). https://doi.org/10.3390/math10091410Samigulina, G.A., Samigulina, Z.I., Bekeshev, D., and Butakova, D. Data-driven machinery faults detection techniques using artificial intelligence in Industry 4.0 concept. Procedia Computer Science, 257, 404–411 (2025). https://doi.org/10.1016/j.procs.2025.03.053Samigulina, G.A., Samigulina, Z.I., Bekeshev, D., and Butakova, D. Data-driven machinery faults detection techniques using artificial intelligence in Industry 4.0 concept. Procedia Computer Science, 257, 404–411 (2025). https://doi.org/10.1016/j.procs.2025.03.053

The authors declare that there are no conflicts of interest present.