A THREE-TIER INFORMATION LIFECYCLE MANAGEMENT MECHANISM FOR HOT-TIER MEMORY CONTROL IN CONTAINERIZED KAFKA–FLINK STREAMING PIPELINES

Real-time streaming systems face persistent memory pressure as continuous data ingestion drives unbounded growth in hot-tier storage. Existing Information Lifecycle Management (ILM) frameworks have been applied primarily to enterprise archival contexts and have not been evaluated within containerized streaming pipelines that employ multi-tier in-memory architectures. This paper presents a lightweight, policy-driven ILM mechanism integrated into a Kafka–Apache Flink pipeline with a three-tier storage model comprising MongoDB (hot-tier), TimescaleDB (warm-tier), and Parquet files (cold-tier). An asynchronous sweeper thread migrates records between tiers according to configurable time thresholds and , preventing hot-tier saturation without disrupting stream processing. Five experiments were conducted to evaluate memory efficiency, tier retrieval latency, threshold sensitivity, scalability, and extended lifecycle behavior. The results demonstrate that ILM reduces MongoDB peak memory usage by 81% (from 106.96 ± 1.63 MB to 20.28 ± 1.81 MB, p < 0.001) while Flink throughput and processing latency remain unaffected. Memory bounds hold stably across ingestion rates from 200 to 1,000 messages per second. An extended 90-minute run validates correct three-tier lifecycle operation, with MongoDB remaining bounded, TimescaleDB absorbing 2.28 million warm records, and Parquet accumulating cold archives. These findings confirm that effective, low-overhead ILM can be achieved in containerized real-time pipelines using only native database capabilities and file system operations.

1. Ding, Y. Research on management and optimization of big data computing engine based on cloud native technology IT architecture. Procedia Computer Science, 243, 910–917 (2024). https://doi.org/10.1016/j.procs.2024.09.109

2. Statista. Volume of data/information created, captured, copied, and consumed worldwide from 2010 to 2025. Statista Research Department (2024). URL: https://www.statista.com/statistics/871513/worldwidedata-created/ (accessed 2026.04.10)

3. Sharma, C., and Vaid, A. Leveraging SAP information lifecycle management (ILM): Latest insights and applications. Zenodo, 5(6), 167–173 (2024).

4. Gandomi, A., and Haider, M. Beyond the hype: Big data concepts, methods, and analytics. International Journal of Information Management, 35, 137–144 (2015).

5. Govil, J., Kaur, N., Kaur, H., and Govil, J. Data/information lifecycle management: A solution for taming data beast. Proceedings of the International Conference on Information Technology: New Generations (ITNG 2008), 1226–1227 (2008).

6. Salutina, T.Y., Klesareva, E.Y., and Platunina, G.P. Big data as a management decision-making tool in digital business environments. Proceedings of the 2021 IEEE International Conference on Quality Management, Transport and Information Security, Information Technologies (IT&QM&IS 2021), 893–895 (2021).

7. Ahmad, P.H., and Rai, M. Analysis of optimization strategies for big data storage management: A study. Proceedings of the 2023 4th International Conference on Electronics and Sustainable Communication Systems (ICESC 2023), 1747–1753 (2023).

8. Wang, M.F., Lin, W.T., Tang, C.H., and Tsai, M.F. Constructing storage capacity migration policies for information lifecycle management system. Proceedings of the 12th IEEE International Conference on High Performance Computing and Communications (HPCC 2010), 503–508 (2010).

9. Li-Zhen, S. Research on hierarchical storage of digital library based on the information lifecycle management. Proceedings of the 2010 2nd IEEE International Conference on Information Management and Engineering, 64–66 (2010).

10. HongJu, X., Fei, W., FenMei, W., and XiuZhen, W. Some key problems of data management in army data engineering based on big data. Proceedings of the 2017 IEEE 2nd International Conference on Big Data Analysis (ICBDA 2017), 149–152 (2017).

11. Miao, T. The optimization of land resource data management model in big data era. Proceedings of the IEEE Advanced Information Technology, Electronic and Automation Control Conference (IAEAC 2024), 763–767 (2024).

12. Muratov, S.Y., and Muravyov, S.B. Framework architecture of a secure big data lake. Procedia Computer Science, 229, 39–46 (2023).

13. Kreps, J., Narkhede, N., and Rao, J. Kafka: A distributed messaging system for log processing. Proceedings of the NetDB Workshop, 11, 1–7 (2011).

14. Carbone, P., Katsifodimos, A., Ewen, S., Markl, V., Haridi, S., and Tzoumas, K. Apache Flink: Stream and batch processing in a single engine. Bulletin of the Technical Committee on Data Engineering, 38(4) (2015).

15. Chintapalli, S., Dagit, D., Evans, B., Farivar, R., Graves, T., Holderbaum, M., and Poulosky, P. Benchmarking streaming computation engines: Storm, Flink and Spark Streaming. Proceedings of the 2016 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW 2016), 1789–1792 (2016).

16. Kulkarni, A., and Freedman, M. TimescaleDB: SQL made scalable for time-series data. Proceedings of the CIDR Workshop (2017). URL: https://api.semanticscholar.org/CorpusID:34446750

17. Apache Parquet Project. Apache Parquet format specification (2024). URL: https://parquet.apache.org/docs/ (accessed: 2026.04.10).