IINTELLIGENT HYBRID METHOD FOR AUTOMATED PENETRATION TESTING IN 5G AND BEYOND NETWORKS

The evolution of 5G and the anticipated introduction of 6G technologies significantly increases network complexity through service-based architecture, network slicing, virtualization and distributed cloud-native functions. These advancements improve scalability and flexibility but simultaneously expand the attack surface and introduce novel vulnerabilities. Traditional penetration testing methodologies are not suitable for such dynamic and virtualized environments because they rely on static procedures and manual testing that cannot match the speed and structural variability of modern mobile networks. In parallel, machine learning–based intrusion detection systems (IDS) demonstrate strong capabilities in detecting anomalous and zero-day behaviors but operate independently from penetration testing processes. This paper presents an intelligent hybrid method for automated penetration testing in 5G and beyond networks, integrating a machine-learning intrusion detection system based on incremental learning, autoencoders and generative adversarial networks (GANs) – with an attack optimization module driven by the Differential Evolution (DE) algorithm. Instead of a genetic algorithm, DE is employed due to its fast convergence, robustness to local minima, and suitability for optimizing high-dimensional representations of attack strategies. An experimental evaluation on a real OpenAirInterface-based 5G Standalone testbed demonstrates that the DEdriven approach improves vulnerability identification efficiency, produces optimal multi-stage attack strategies, and enables realistic automated penetration scenarios. These results indicate that DE-based optimization provides a scalable, adaptive and efficient foundation for continuous security assessment of next-generation mobile networks.

1. 3GPP. System Architecture for the 5G System (5GS). 3GPP TS 23.501, Release 19 (2025).

2. 3GPP. Security Architecture and Procedures for 5G System. 3GPP TS 33.501, Release 19 (2025).

3. 3GPP. Management and Orchestration: Concepts, Use Cases and Architecture. 3GPP TS 28.530, Release 19 (2025).

4. Singh, A., Juneja, D., Singh, J., and Aggarwal, S. Security Technologies for 6G Mobile Systems and Challenges Associated with AI Technologies. Lecture Notes in Electrical Engineering, 1444, 1–13 (2026). https://doi.org/10.1007/978-981-96-8283-6_1

5. Albrecht, M.R., and Jensen, R.B. The Vacuity of the Open Source Security Testing Methodology Manual. Lecture Notes in Computer Science, 12529, 114–147 (2020). https://doi.org/10.1007/978-3-030-64357-7_6

6. Lidanta, F.Z., Almaarif, A., and Budiyono, A. Vulnerability Analysis of Wireless LAN Networks Using Penetration Testing Execution Standard: A Case Study of Cafes in Palembang. Proceedings of the 2021 International Conference on ICT for Smart Society (ICISS), 1–5 (2021). https://doi.org/10.1109/ICISS53185.2021.9533216

7. Scarfone, K., Souppaya, M., Cody, A., and Orebaugh, A. Technical Guide to Information Security Testing and Assessment. NIST Special Publication 800-115 (2008). https://doi.org/10.6028/NIST.SP.800-115

8. OWASP Foundation. OWASP Web Security Testing Guide v4 (2021). https://owasp.org/www-projectweb-security-testing-guide/v41/

9. Harvanek, M., Bolcek, J., Kufa, J., Polak, L., Simka, M., and Marsalek, R. Survey on 5G Physical Layer Security Threats and Countermeasures. Sensors, 24, 5523 (2024). https://doi.org/10.3390/s24175523

10. Imanbayev, A., Tynymbayev, S., Odarchenko, R., Gnatyuk, S., Berdibayev, R., Baikenov, A., and Kaniyeva, N. Research of Machine Learning Algorithms for the Development of Intrusion Detection Systems in 5G Mobile Networks and Beyond. Sensors, 22 (24), 9957 (2022). https://doi.org/10.3390/s22249957

11. Pinchuk, A., Odarchenko, R., Samoilenko, V., and Imanbayev, A. 5G Network Deployment Based on Open-source Projects: A Comparative Analysis. Proceedings of the IEEE International Conference on Intelligent Data Acquisition and Advanced Computing Systems (IDAACS), 596–601 (2023). https://doi.org/10.1109/IDAACS58523.2023.10348675

12. Storn, R., and Price, K. Differential Evolution – A Simple and Efficient Heuristic for Global Optimization over Continuous Spaces. Journal of Global Optimization, 11, 341–359 (1997). https://doi.org/10.1023/A:1008202821328

13. Almutairi, M.S. Deep Learning-Based Solutions for 5G Network and 5G-Enabled Internet of Vehicles: Advances, Meta-Data Analysis, and Future Direction. Mathematical Problems in Engineering, Article ID 6855435 (2022). https://doi.org/10.1155/2022/6855435

14. Dai, W., Zhou, J., Ye, C., and Xu, G. Automated Penetration Testing System Based on PTES and ATT&CK. Communications in Computer and Information Science, 2594, 433–444 (2026). https://doi.org/10.1007/978-981-95-1340-6_35

15. Odarchenko, R., Iavich, M., and Pinchuk, A. Development of a Method for Automated 5G and Beyond Network Slices Penetration Testing. Radioelectronic and Computer Systems, 1 (113), 248–263 (2025). https:// doi.org/10.32620/reks.2025.1.17

16. Allaw, Z., Zein, O., and Ahmad, A.-M. Cross-Layer Security for 5G/6G Network Slices: An SDN, NFV, and AI-Based Hybrid Framework. Sensors, 25 (11), 3335 (2025). https://doi.org/10.3390/s25113335

17. Eltaeib, T., and Mahmood, A. Differential Evolution: A Survey and Analysis. Applied Sciences, 8 (10), 1945 (2018). https://doi.org/10.3390/app8101945

18. Noor, K., Imoize, A.L., Li, C.-T., and Weng, C.-Y. A Review of Machine Learning and Transfer Learning Strategies for Intrusion Detection Systems in 5G and Beyond. Mathematics, 13 (7), 1088 (2025). https://doi.org/10.3390/math13071088