kaz29Вестник Казахстанско-Британского технического университетаHerald of the Kazakh-British Technical University1998-66882959-8109Казахстанско-Британский Технический Университет10.55452/1998-6688-2025-22-1-25-35kaz29-1727Research ArticleКОМПЬЮТЕРНЫЕ НАУКИCOMPUTER SCIENCEМЕТОДЫ ОБРАБОТКИ И АНАЛИЗА БОЛЬШИХ ДАННЫХ: ПОДХОДЫ И ПЕРСПЕКТИВЫ В ЗАДАЧАХ МАШИННОГО ОБУЧЕНИЯMETHODS OF PROCESSING AND ANALYZING BIG DATA IN MACHINE LEARNING TASKS: APPROACHES AND PROSPECTShttps://orcid.org/0009-0004-5261-2700КомаровН.KomarovN.

 магистрант

 г. Алматы 

 Master’s student 

 Almaty 

41384@iitu.edu.kzhttps://orcid.org/0000-0001-8761-4272МухановС. Б.MukhanovS. B.

 PhD, ассистент-профессор 

 г. Алматы 

 PhD, assistant professor 

 Almaty 

s.mukhanov@iitu.edu.kzhttps://orcid.org/0009-0001-1917-169XБазарбековИ. М.BazarbekovI. M.

 магистр, сениор-лектор 

 г. Алматы 

 Master, senior-lecturer 

 Almaty 

i.bazarbekov@iitu.edu.kzhttps://orcid.org/0000-0001-9112-5922ЖакыпбековС. Ж.ZhakypbekovS. Zh.

 магистр, сениор-лектор 

 г. Алматы 

 Master, senior-lecturer 

 Almaty 

s.zhakypbekov@iitu.edu.kzhttps://orcid.org/0000-0002-6502-8907СибанбаеваС. Е.SibanbayevaS. Y.

 PhD, ассистент-профессор 

 г. Алматы 

 PhD, Assistant Professor 

 Almaty 

s.sibanbayeva@almau.edu.kzМеждународный университет информационных технологийКазахстанInternational University of Information TechnologiesKazakhstanАлматы Менеджмент УниверситетКазахстанAlmaty Management UniversityKazakhstan2025220320252212535Copyright © Комаров Н., Муханов С.Б., Базарбеков И.М., Жакыпбеков С.Ж., Сибанбаева С.Е., 20252025Комаров Н., Муханов С.Б., Базарбеков И.М., Жакыпбеков С.Ж., Сибанбаева С.Е.Komarov N., Mukhanov S.B., Bazarbekov I.M., Zhakypbekov S.Z., Sibanbayeva S.Y.Данная работа распространяется под лицензией Creative Commons Attribution 4.0.This work is licensed under a Creative Commons Attribution 4.0 License.https://vestnik.kbtu.edu.kz/jour/article/view/1727

В статье рассматриваются методы обработки и анализа больших данных с целью повышения точности и эффективности моделей машинного обучения (МО). Основное внимание уделено задачам классификации, эффективности алгоритмов, таких как XGBoost, метод опорных векторов (SVM), ансамблевые методы, а также системам работы с большими данными, включая Hadoop и Apache Spark. Описаны ключевые этапы работы с данными: очистка, нормализация, выбор признаков, что критически важно для построения устойчивых моделей. Для оценки эффективности алгоритмов использовались метрики точности, полноты, F-меры и AUC-ROC. Особое внимание уделено применению МО в контексте организационных инноваций. Рассмотрены перспективы интеграции распределенных вычислительных платформ с алгоритмами МО.

This article explores the methods of processing and analyzing big data in order to improve the accuracy and efficiency of machine learning (MO) models. The main focus is on classification problems, the effectiveness of algorithms such as XGBoost, the support vector machine (SVM), ensemble methods, as well as systems for working with big data, including Hadoop and Apache Spark. The key stages of working with data are described: cleaning, normalization, selection of features, which is critically important for building stable models on large amounts of data. Accuracy, completeness, F-measure, and AUC-ROC metrics were used to evaluate the effectiveness of the algorithms, which made it possible to conduct a comparative analysis and identify the most productive approaches. Special attention is paid to the application of MO in the context of organizational innovations, including the tasks of classification, forecasting the success of innovations and innovation portfolio management. Recommendations on the choice of technologies and algorithms for various data types and scales are presented, and prospects for integrating distributed computing platforms with MO algorithms to achieve scalable and efficient solutions are discussed.

большие данныеобработка данныхмашинное обучениеApache SparkHadoopXGBoostметод опорных векторовклассификацияоптимизация данныхраспределенные вычисленияметрики оценки моделейискусственный интеллектинновационные процессыBig datadata processingmachine learningApache SparkHadoopXGBoostsupport vector machine (SVM) methodensemble methodsclassificationdata optimizationdistributed computingmodel evaluation metricsartificial intelligenceinnovative processesinnovative developmenttypes of machine learningReferencesJunfei Qiu, Qihui Wu, Guoru Ding, Yuhua Xu & Shuo Feng. A survey of machine learning for big data processing. EURASIP Journal on Advances in Signal Processing, 2016, vol. 2016, article no. 67.Junfei Qiu, Qihui Wu, Guoru Ding, Yuhua Xu & Shuo Feng. A survey of machine learning for big data processing. EURASIP Journal on Advances in Signal Processing, 2016, vol. 2016, article no. 67.Najafabadi M.M., Villanustre F., Khoshgoftaar T.M. et al. Deep learning applications and challenges in big data analytics. Journal of Big Data, 2015, no. 2, article no. 1. https://doi.org/10.1186/s40537-014-0007-7.Najafabadi M.M., Villanustre F., Khoshgoftaar T.M. et al. Deep learning applications and challenges in big data analytics. Journal of Big Data, 2015, no. 2, article no. 1. https://doi.org/10.1186/s40537-014-0007-7.Järvinen P., Siltanen P., Kirschenbaum A. Data Analytics and Machine Learning. In: Södergård C., Mildorf T., Habyarimana E., Berre A.J., Fernandes J.A., Zinke-Wehlmann C. (eds) Big Data in Bioeconomy. Springer, Cham. 2021. https://doi.org/10.1007/978-3-030-71069-9_10.Järvinen P., Siltanen P., Kirschenbaum A. Data Analytics and Machine Learning. In: Södergård C., Mildorf T., Habyarimana E., Berre A.J., Fernandes J.A., Zinke-Wehlmann C. (eds) Big Data in Bioeconomy. Springer, Cham. 2021. https://doi.org/10.1007/978-3-030-71069-9_10.Sarker I.H. Machine Learning: Algorithms, Real-World Applications and Research Directions. SN COMPUT. SCI., 2021, vol. 2, article no. 160. https://doi.org/10.1007/s42979-021-00592-x.Sarker I.H. Machine Learning: Algorithms, Real-World Applications and Research Directions. SN COMPUT. SCI., 2021, vol. 2, article no. 160. https://doi.org/10.1007/s42979-021-00592-x.Dastan Hussen Maulud, Adnan Mohsin Abdulazeez. A Review on Linear Regression Comprehensive in Machine Learning, Journal of Applied Science and Technology Trends, 2020, vol. 1, no. 4, pp. 140–147. https://doi.org/10.38094/jastt1457.Dastan Hussen Maulud, Adnan Mohsin Abdulazeez. A Review on Linear Regression Comprehensive in Machine Learning, Journal of Applied Science and Technology Trends, 2020, vol. 1, no. 4, pp. 140–147. https://doi.org/10.38094/jastt1457.Maher Maalouf. Logistic regression in data analysis: An overview.International Journal of Data Analysis, Techniques and Strategies, 2011, vol. 3, no. 3, pp. 281–299, https://doi.org/10.1504/IJDATS.2011.041335.Maher Maalouf. Logistic regression in data analysis: An overview.International Journal of Data Analysis, Techniques and Strategies, 2011, vol. 3, no. 3, pp. 281–299, https://doi.org/10.1504/IJDATS.2011.041335.Ivanov A.A. Iskusstvennyj intellekt kak osnova innovacionnyh preobrazovanij v tehnike, jekonomike, biznese. Izvestija SPbGJeU, 2018, no. 3 (111), pp. 112–115. [in Russian]Ivanov A.A. Iskusstvennyj intellekt kak osnova innovacionnyh preobrazovanij v tehnike, jekonomike, biznese. Izvestija SPbGJeU, 2018, no. 3 (111), pp. 112–115. [in Russian]Sejdametova Z.S. Jekonomika i mashinnoe obuchenie. Uchenye zapiski Krymskogo inzhenernopedagogicheskogo universiteta, 2019, no. 1(63), pp. 167–171. [in Russian]Sejdametova Z.S. Jekonomika i mashinnoe obuchenie. Uchenye zapiski Krymskogo inzhenernopedagogicheskogo universiteta, 2019, no. 1(63), pp. 167–171. [in Russian]Terehov V.I. Metodika podgotovki dannyh dlja obrabotki impul'snymi nejronnymi setjami. Nejrokomp'jutery: razrabotka, primenenie, 2017, no. 2, pp. 31–36. [in Russian]Terehov V.I. Metodika podgotovki dannyh dlja obrabotki impul'snymi nejronnymi setjami. Nejrokomp'jutery: razrabotka, primenenie, 2017, no. 2, pp. 31–36. [in Russian]Flah P. Mashinnoe obuchenie. Nauka i iskusstvo postroenija algoritmov, kotorye izvlekajut znanija iz dannyh, 2015, p. 400. [in Russian]Flah P. Mashinnoe obuchenie. Nauka i iskusstvo postroenija algoritmov, kotorye izvlekajut znanija iz dannyh, 2015, p. 400. [in Russian]Mukhanov S.B., Uskenbayeva R.K. Pattern Recognition with Using Effective Algorithms and Methods of Computer Vision Library. Advances in Intelligent Systems and Computing, 2020, no. 1, pp. 31–37.Mukhanov S.B., Uskenbayeva R.K. Pattern Recognition with Using Effective Algorithms and Methods of Computer Vision Library. Advances in Intelligent Systems and Computing, 2020, no. 1, pp. 31–37.Mukhanov S., Uskenbayeva R., Im Cho Young, Dauren K., Les N., Amangeldi M. Gesture Recognition of Machine Learning and Convolutional Neural Network Methods for Kazakh Sign Language. Herald Scientific Journal of Astana IT University, 2023, vol. 15, pp. 16–27.Mukhanov S., Uskenbayeva R., Im Cho Young, Dauren K., Les N., Amangeldi M. Gesture Recognition of Machine Learning and Convolutional Neural Network Methods for Kazakh Sign Language. Herald Scientific Journal of Astana IT University, 2023, vol. 15, pp. 16–27.Mukhanov S.B., Lee A.S., Zheksenov D.B., Yevdokimov D.D., Amirgaliev E.N., Kalzhigitov N.K., Kenshimov Sh. Comparative analysis of neural network models for gesture recognition methods hands. Bulletin of NIA RK. Information and communication technologies, 2023, no. 2(88), pp. 15–27.Mukhanov S.B., Lee A.S., Zheksenov D.B., Yevdokimov D.D., Amirgaliev E.N., Kalzhigitov N.K., Kenshimov Sh. Comparative analysis of neural network models for gesture recognition methods hands. Bulletin of NIA RK. Information and communication technologies, 2023, no. 2(88), pp. 15–27.Kenshimov C., Mukhanov S., Merembayev T., Yedilkhan D. A Comparison of Convolutional Neural Networks for Kazakh Sign Language Recognition Eastern-European. Journal of Enterprise Technologies, 2021, vol. 5, no. 2–113, pp. 44–54.Kenshimov C., Mukhanov S., Merembayev T., Yedilkhan D. A Comparison of Convolutional Neural Networks for Kazakh Sign Language Recognition Eastern-European. Journal of Enterprise Technologies, 2021, vol. 5, no. 2–113, pp. 44–54.

The authors declare that there are no conflicts of interest present.