REINFORCEMENT LEARNING–DRIVEN CONTROL STRATEGIES FOR SMART MANUFACTURING SYSTEMS

Nowadays, the creation of smart manufacturing systems has high importance. Neural networks have been widely applied to solve complex manufacturing challenges. The paper is devoted to the study of neural networks with reinforcement learning as PPO (Proximal Policy Optimization), DQN (Deep Q-network) for state diagnosis of industrial equipment within the GEMMA (Guide d’Etude des Modes de Marche er d’Arret) model. The GEMMA French approach is established on the SFC (Sequential Function Charts) language and includes standards for controlling technical processes. An application of neural networks in area D of the GEMMA model is introduced. Modelling and experimental results were conducted based on synthetic and experimental datasets. The implementation of the architecture considered allows us to achieve reliable results for industrial data.

1. Jayakumar, S., and Nandakumar, S. Distributed resource optimization using the Q-learning algorithm in device-to-device communication: A reinforcement learning paradigm. Results in Engineering, 23, 102462 (2024). https://doi.org/10.1016/j.rineng.2024.102462

2. Xia, K., Sacco, C., Kirkpatrick, M., et al. A digital twin to train deep reinforcement learning agent for smart manufacturing plants: Environment, interfaces and intelligence. Journal of Manufacturing Systems, 58, 210–230 (2021). https://doi.org/10.1016/j.jmsy.2020.06.012

3. Park, K.T., Son, Y.H., Ko, S.W., and Noh, S.D. Digital twin and reinforcement learning-based resilient production control for micro smart factory. Applied Sciences, 11 (7) (2021). https://doi.org/10.3390/app11072977

4. Ullah, I., Hassan, U.U., and Ali, M.I. Multi-level federated learning for Industry 4.0: A crowdsourcing approach. Procedia Computer Science, 217, 423–435 (2022). https://doi.org/10.1016/j.procs.2022.12.238

5. Mezair, T., Djenouri, Y., and Belhadi, A. A sustainable deep learning framework for fault detection in 6G Industry 4.0 heterogeneous data environments. Computer Communications, 187, 164–171 (2022). https://doi.org/10.1016/j.comcom.2022.02.010

6. Neupane, D., and Seok, J. Bearing fault detection and diagnosis using Case Western Reserve University dataset with deep learning approaches: A review. IEEE Access, 8, 93155–93178 (2020). https://doi.org/10.1109/ACCESS.2020.2990528

7. Mattera, G., Mattera, R., Vespoli, S., and Salatiello, E. Anomaly detection in manufacturing systems with temporal networks and unsupervised machine learning. Computers & Industrial Engineering, 203, 111023 (2025). https://doi.org/10.1016/j.cie.2025.111023

8. Prabu, S., Senthilraja, R., Ali, A.M., Jayapoorani, S., and Arun, M. AI-driven predictive maintenance for smart manufacturing systems using digital twin technology. International Journal of Computational and Experimental Science and Engineering, 11 (1), 1350–1355 (2025). https://doi.org/10.22399/ijcesen.1099

9. Asmat, H., Din, I., Almogren, A., and Khan, M. Digital twin with soft actor-critic reinforcement learning for transitioning from Industry 4.0 to 5.0. IEEE Access (2025). https://doi.org/10.1109/ACCESS.2025.3546085

10. Ruiz Rodríguez, M.L., Kubler, S., de Giorgio, A., Cordy, M., Robert, J., and Le Traon, Y. Multi-agent deep reinforcement learning based predictive maintenance on parallel machines. Robotics and Computer-Integrated Manufacturing, 78, 102406 (2022). https://doi.org/10.1016/j.rcim.2022.102406

11. Pliego Marugán, A., Pinar-Pérez, J.M., and García Márquez, F.P. A reinforcement learning agent for maintenance of deteriorating systems with increasingly imperfect repairs. Reliability Engineering & System Safety, 252, 110466 (2024). https://doi.org/10.1016/j.ress.2024.110466

12. Serrano-Ruiz, J.C., Mula, J., and Poler, R. Development of a multidimensional conceptual model for job shop smart manufacturing scheduling from the Industry 4.0 perspective. Journal of Manufacturing Systems, 63, 185–202 (2022). https://doi.org/10.1016/j.jmsy.2022.03.011

13. Zhou, L., Zhang, L., and Horn, B. Deep reinforcement learning-based dynamic scheduling in smart manufacturing. Procedia CIRP, 93, 383–388 (2020). https://doi.org/10.1016/j.procir.2020.05.163

14. Zhang, M., Lu, Y., Hu, Y., et al. Dynamic scheduling method for job-shop manufacturing systems by deep reinforcement learning with proximal policy optimization. Sustainability, 14 (9) (2022). https://doi.org/10.3390/su14095177

15. Yang, S., and Xu, Z. Intelligent scheduling and reconfiguration via deep reinforcement learning in smart manufacturing. International Journal of Production Research, 60 (16), 4936–4953 (2022). https://doi.org/10.1080/00207543.2021.1943037

16. Ho, G., Tang, Y., Leung, E., and Tong, P. Integrated reinforcement learning of automated guided vehicles dynamic path planning for smart logistics and operations. Transportation Research Part E: Logistics and Transportation Review, 196 (2025). https://doi.org/10.1016/j.tre.2025.104008

17. Zhou, T., Tang, D., and Wang, L. Reinforcement learning with composite rewards for production scheduling in a smart factory. IEEEAccess, 9, 752–766 (2021). https://doi.org/10.1109/ACCESS.2020.3046784

18. Bin Shaikat, F., and Ahmed Faysal, S. Optimization of production scheduling in smart manufacturing environments using machine learning algorithms (2025).

19. Kim, Y., Kim, H., Ha, B., and Kim, W. Federated digital twins: A scheduling approach based on temporal graph neural network and deep reinforcement learning. IEEE Access (2025). https://doi.org/10.1109/ACCESS.2025.3530558

20. Chen, Y., Zhang, J., and Rauf, M. Dynamic scheduling of hybrid flow shop problem with uncertain process time and flexible maintenance using NeuroEvolution of Augmenting Topologies. IET Collaborative Intelligent Manufacturing, 6 (3) (2024). https://doi.org/10.1049/cim2.12119

21. Schoepp, S., Taghian, M., and Miwa, S. Enhancing hardware fault tolerance in machines with reinforcement learning policy gradient algorithms (2024). https://doi.org/10.48550/arXiv.2407.15283

22. Modirrousta, M.H., Aliyari Shoorehdeli, M., and Yari, M. Imbalanced classification in faulty turbine data: New proximal policy optimisation. IET Collaborative Intelligent Manufacturing, 6 (3) (2024). https://doi.org/10.1049/cim2.12114

23. Yang, Y., Juntao, L., and Lingling, P. Multi-robot path planning based on a deep reinforcement learning DQN algorithm. CAAI Transactions on Intelligence Technology, 5 (3), 177–183 (2020). https://doi.org/10.1049/trit.2020.0024

24. Zhu, Q., Han, J., Chai, K., and Zhao, C. Time series analysis based on Informer algorithms: A survey. Symmetry, 15 (4) (2023). https://doi.org/10.3390/sym15040951

25. Zhang, X., Yang, K., and Zheng, L. Transformer fault diagnosis method based on TimesNet and Informer. Actuators, 13 (2) (2024). https://doi.org/10.3390/act13020074

26. Pan, Y., et al. Understanding and mitigating the limitations of prioritized experience replay (2023). https://doi.org/10.48550/arXiv.2007.09569

27. Bu, F., and Chang, D.E. Double prioritized state recycled experience replay. In: Proceedings of the IEEE International Conference on Consumer Electronics Asia (2020). https://doi.org/10.1109/ICCEAsia49877.2020.9276975

28. Chandrinos, N., Loi, I., Zachos, P., et al. Effectiveness of L2 regularization in privacy-preserving machine learning (2024). https://doi.org/10.48550/arXiv.2412.01541

29. Alinda Rahmi, N., and Defit, S. The use of hyperparameter tuning in model classification: A scientific work area identification. International Journal of Applied Sciences and Smart Technologies, 8 (4) (2025). http://dx.doi.org/10.62527/joiv.8.4.3092

30. Shekhar, S., Bansode, A., and Salim, A. A comparative study of hyper-parameter optimization tools. In: 2021 IEEE Asia-Pacific Conference on Computer Science and Data Engineering (CSDE) (2021). https://doi.org/10.1109/CSDE53843.2021.9718485

31. Jordon, J., Szpruch, L., Houssiau, F., et al. Synthetic data – what, why and how? (2022). https://doi.org/10.48550/arXiv.2205.03257

32. Fan, C., Chen, M., Wang, X., et al. A review on data preprocessing techniques toward efficient and reliable knowledge discovery from building operational data. Frontiers in Energy Research, 9 (2021). https://doi.org/10.3389/fenrg.2021.652801

33. Bekar, E.T., Nyqvist, P., and Skoogh, A. An intelligent approach for data pre-processing and analysis in predictive maintenance with an industrial case study. Advances in Mechanical Engineering, 12 (5) (2020). https://doi.org/10.1177/168781402091

34. Ou, F., Wang, H., Zhang, C., et al. Industrial data-driven machine learning soft sensing for optimal operation of etching tools. Digital Chemical Engineering, 13 (2024). https://doi.org/10.1016/j.dche.2024.100195

35. Terven, J., Cordova-Esparza, D., Ramirez-Pedraza, A., et al. Loss functions and metrics in deep learning (2023). https://doi.org/10.48550/arXiv.2307.02694

36. Le, P.B., and Nguyen, Z.T. ROC curves, loss functions, and distorted probabilities in binary classification. Mathematics, 10 (9) (2022). https://doi.org/10.3390/math10091410

37. Samigulina, G.A., Samigulina, Z.I., Bekeshev, D., and Butakova, D. Data-driven machinery faults detection techniques using artificial intelligence in Industry 4.0 concept. Procedia Computer Science, 257, 404–411 (2025). https://doi.org/10.1016/j.procs.2025.03.053