kaz29Вестник Казахстанско-Британского технического университетаHerald of the Kazakh-British Technical University1998-66882959-8109Казахстанско-Британский Технический Университет10.55452/1998-6688-2026-23-2-159-170kaz29-2894Research ArticleКОМПЬЮТЕРНЫЕ НАУКИCOMPUTER SCIENCEТРЕХУРОВНЕВЫЙ МЕХАНИЗМ УПРАВЛЕНИЯ ЖИЗНЕННЫМ ЦИКЛОМ ИНФОРМАЦИИ ДЛЯ КОНТРОЛЯ ПАМЯТИ ГОРЯЧЕГО УРОВНЯ В КОНТЕЙНЕРИЗОВАННЫХ СТРИМИНГ-ПАЙПЛАЙНАХ KAFKA–FLINKA THREE-TIER INFORMATION LIFECYCLE MANAGEMENT MECHANISM FOR HOT-TIER MEMORY CONTROL IN CONTAINERIZED KAFKA–FLINK STREAMING PIPELINEShttps://orcid.org/0009-0004-4379-1065ОңаеваА. Е.OnayevaA. E.

Магистр.

Алмат

MSc.

Almaty

al_onayeva@kbtu.kzКазахстанско-Британский технический университетКазахстанKazakh-British Technical UniversityKazakhstan202627062026232159170Copyright © Оңаева А.Е., 20262026Оңаева А.Е.Onayeva A.E.Данная работа распространяется под лицензией Creative Commons Attribution 4.0.This work is licensed under a Creative Commons Attribution 4.0 License.https://vestnik.kbtu.edu.kz/jour/article/view/2894

Системы потоковой обработки в реальном времени сталкиваются с постоянной нагрузкой на память, поскольку непрерывное поступление данных приводит к неограниченному росту хранилища горячего уровня. Существующие подходы управления жизненным циклом информации (Information Lifecycle Management, ILM) в основном применялись в корпоративных системах архивирования и не были исследованы в контексте контейнеризованных стриминговых пайплайнов, использующих многоуровневые архитектуры хранения в памяти. В данной работе представлен легкий механизм ILM, управляемый политиками и интегрированный в пайплайн Kafka–Apache Flink с трехуровневой моделью хранения: MongoDB (горячий уровень), TimescaleDB (теплый уровень) и файлы Parquet (холодный уровень). Асинхронный поток очистки (sweeper thread) перемещает записи между уровнями в соответствии с настраиваемыми временными порогами и , предотвращая переполнение горячего уровня без нарушения потоковой обработки. Было проведено пять экспериментов для оценки эффективности использования памяти, задержки доступа к данным по уровням, чувствительности к пороговым значениям, масштабируемости и поведения при длительной эксплуатации. Результаты показывают, что применение ILM снижает пиковое использование памяти MongoDB на 81% (с 106.96 ± 1.63 МБ до 20.28 ± 1.81 МБ, p < 0.001), при этом пропускная способность и задержка обработки в Flink остаются неизменными. Ограничения по памяти стабильно соблюдаются при скорости поступления данных от 200 до 1000 сообщений в секунду. Дополнительный эксперимент продолжительностью 90 минут подтверждает корректную работу трехуровневого жизненного цикла: MongoDB остается в заданных пределах, TimescaleDB аккумулирует 2.28 миллиона записей теплого уровня, а Parquet формирует холодный архив. Полученные результаты подтверждают, что эффективное и малозатратное управление жизненным циклом данных возможно в контейнеризованных системах потоковой обработки в реальном времени с использованием только встроенных возможностей баз данных и операций файловой системы.

Real-time streaming systems face persistent memory pressure as continuous data ingestion drives unbounded growth in hot-tier storage. Existing Information Lifecycle Management (ILM) frameworks have been applied primarily to enterprise archival contexts and have not been evaluated within containerized streaming pipelines that employ multi-tier in-memory architectures. This paper presents a lightweight, policy-driven ILM mechanism integrated into a Kafka–Apache Flink pipeline with a three-tier storage model comprising MongoDB (hot-tier), TimescaleDB (warm-tier), and Parquet files (cold-tier). An asynchronous sweeper thread migrates records between tiers according to configurable time thresholds and , preventing hot-tier saturation without disrupting stream processing. Five experiments were conducted to evaluate memory efficiency, tier retrieval latency, threshold sensitivity, scalability, and extended lifecycle behavior. The results demonstrate that ILM reduces MongoDB peak memory usage by 81% (from 106.96 ± 1.63 MB to 20.28 ± 1.81 MB, p < 0.001) while Flink throughput and processing latency remain unaffected. Memory bounds hold stably across ingestion rates from 200 to 1,000 messages per second. An extended 90-minute run validates correct three-tier lifecycle operation, with MongoDB remaining bounded, TimescaleDB absorbing 2.28 million warm records, and Parquet accumulating cold archives. These findings confirm that effective, low-overhead ILM can be achieved in containerized real-time pipelines using only native database capabilities and file system operations.

управление жизненным циклом информации (ILM)Apache KafkaApache FlinkMongoDBTimescaleDBпотоковая обработка в реальном времениоптимизация памятимногоуровневое хранениеконтейнеризованные пайплайныархивирование данныхInformation Lifecycle Management (ILM)Apache KafkaApache FlinkMongoDBTimescaleDBreal-time streamingmemory optimizationtiered storagecontainerized pipelinesdata archivingReferencesDing, Y. Research on management and optimization of big data computing engine based on cloud native technology IT architecture. Procedia Computer Science, 243, 910–917 (2024). https://doi.org/10.1016/j.procs.2024.09.109Ding, Y. Research on management and optimization of big data computing engine based on cloud native technology IT architecture. Procedia Computer Science, 243, 910–917 (2024). https://doi.org/10.1016/j.procs.2024.09.109Statista. Volume of data/information created, captured, copied, and consumed worldwide from 2010 to 2025. Statista Research Department (2024). URL: https://www.statista.com/statistics/871513/worldwidedata-created/ (accessed 2026.04.10)Statista. Volume of data/information created, captured, copied, and consumed worldwide from 2010 to 2025. Statista Research Department (2024). URL: https://www.statista.com/statistics/871513/worldwidedata-created/ (accessed 2026.04.10)Sharma, C., and Vaid, A. Leveraging SAP information lifecycle management (ILM): Latest insights and applications. Zenodo, 5(6), 167–173 (2024).Sharma, C., and Vaid, A. Leveraging SAP information lifecycle management (ILM): Latest insights and applications. Zenodo, 5(6), 167–173 (2024).Gandomi, A., and Haider, M. Beyond the hype: Big data concepts, methods, and analytics. International Journal of Information Management, 35, 137–144 (2015).Gandomi, A., and Haider, M. Beyond the hype: Big data concepts, methods, and analytics. International Journal of Information Management, 35, 137–144 (2015).Govil, J., Kaur, N., Kaur, H., and Govil, J. Data/information lifecycle management: A solution for taming data beast. Proceedings of the International Conference on Information Technology: New Generations (ITNG 2008), 1226–1227 (2008).Govil, J., Kaur, N., Kaur, H., and Govil, J. Data/information lifecycle management: A solution for taming data beast. Proceedings of the International Conference on Information Technology: New Generations (ITNG 2008), 1226–1227 (2008).Salutina, T.Y., Klesareva, E.Y., and Platunina, G.P. Big data as a management decision-making tool in digital business environments. Proceedings of the 2021 IEEE International Conference on Quality Management, Transport and Information Security, Information Technologies (IT&QM&IS 2021), 893–895 (2021).Salutina, T.Y., Klesareva, E.Y., and Platunina, G.P. Big data as a management decision-making tool in digital business environments. Proceedings of the 2021 IEEE International Conference on Quality Management, Transport and Information Security, Information Technologies (IT&QM&IS 2021), 893–895 (2021).Ahmad, P.H., and Rai, M. Analysis of optimization strategies for big data storage management: A study. Proceedings of the 2023 4th International Conference on Electronics and Sustainable Communication Systems (ICESC 2023), 1747–1753 (2023).Ahmad, P.H., and Rai, M. Analysis of optimization strategies for big data storage management: A study. Proceedings of the 2023 4th International Conference on Electronics and Sustainable Communication Systems (ICESC 2023), 1747–1753 (2023).Wang, M.F., Lin, W.T., Tang, C.H., and Tsai, M.F. Constructing storage capacity migration policies for information lifecycle management system. Proceedings of the 12th IEEE International Conference on High Performance Computing and Communications (HPCC 2010), 503–508 (2010).Wang, M.F., Lin, W.T., Tang, C.H., and Tsai, M.F. Constructing storage capacity migration policies for information lifecycle management system. Proceedings of the 12th IEEE International Conference on High Performance Computing and Communications (HPCC 2010), 503–508 (2010).Li-Zhen, S. Research on hierarchical storage of digital library based on the information lifecycle management. Proceedings of the 2010 2nd IEEE International Conference on Information Management and Engineering, 64–66 (2010).Li-Zhen, S. Research on hierarchical storage of digital library based on the information lifecycle management. Proceedings of the 2010 2nd IEEE International Conference on Information Management and Engineering, 64–66 (2010).HongJu, X., Fei, W., FenMei, W., and XiuZhen, W. Some key problems of data management in army data engineering based on big data. Proceedings of the 2017 IEEE 2nd International Conference on Big Data Analysis (ICBDA 2017), 149–152 (2017).HongJu, X., Fei, W., FenMei, W., and XiuZhen, W. Some key problems of data management in army data engineering based on big data. Proceedings of the 2017 IEEE 2nd International Conference on Big Data Analysis (ICBDA 2017), 149–152 (2017).Miao, T. The optimization of land resource data management model in big data era. Proceedings of the IEEE Advanced Information Technology, Electronic and Automation Control Conference (IAEAC 2024), 763–767 (2024).Miao, T. The optimization of land resource data management model in big data era. Proceedings of the IEEE Advanced Information Technology, Electronic and Automation Control Conference (IAEAC 2024), 763–767 (2024).Muratov, S.Y., and Muravyov, S.B. Framework architecture of a secure big data lake. Procedia Computer Science, 229, 39–46 (2023).Muratov, S.Y., and Muravyov, S.B. Framework architecture of a secure big data lake. Procedia Computer Science, 229, 39–46 (2023).Kreps, J., Narkhede, N., and Rao, J. Kafka: A distributed messaging system for log processing. Proceedings of the NetDB Workshop, 11, 1–7 (2011).Kreps, J., Narkhede, N., and Rao, J. Kafka: A distributed messaging system for log processing. Proceedings of the NetDB Workshop, 11, 1–7 (2011).Carbone, P., Katsifodimos, A., Ewen, S., Markl, V., Haridi, S., and Tzoumas, K. Apache Flink: Stream and batch processing in a single engine. Bulletin of the Technical Committee on Data Engineering, 38(4) (2015).Carbone, P., Katsifodimos, A., Ewen, S., Markl, V., Haridi, S., and Tzoumas, K. Apache Flink: Stream and batch processing in a single engine. Bulletin of the Technical Committee on Data Engineering, 38(4) (2015).Chintapalli, S., Dagit, D., Evans, B., Farivar, R., Graves, T., Holderbaum, M., and Poulosky, P. Benchmarking streaming computation engines: Storm, Flink and Spark Streaming. Proceedings of the 2016 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW 2016), 1789–1792 (2016).Chintapalli, S., Dagit, D., Evans, B., Farivar, R., Graves, T., Holderbaum, M., and Poulosky, P. Benchmarking streaming computation engines: Storm, Flink and Spark Streaming. Proceedings of the 2016 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW 2016), 1789–1792 (2016).Kulkarni, A., and Freedman, M. TimescaleDB: SQL made scalable for time-series data. Proceedings of the CIDR Workshop (2017). URL: https://api.semanticscholar.org/CorpusID:34446750Kulkarni, A., and Freedman, M. TimescaleDB: SQL made scalable for time-series data. Proceedings of the CIDR Workshop (2017). URL: https://api.semanticscholar.org/CorpusID:34446750Apache Parquet Project. Apache Parquet format specification (2024). URL: https://parquet.apache.org/docs/ (accessed: 2026.04.10).Apache Parquet Project. Apache Parquet format specification (2024). URL: https://parquet.apache.org/docs/ (accessed: 2026.04.10).

The authors declare that there are no conflicts of interest present.