kaz29Вестник Казахстанско-Британского технического университетаHerald of the Kazakh-British Technical University1998-66882959-8109Казахстанско-Британский Технический Университет10.55452/1998-6688-2026-23-2-218-232kaz29-2902Research ArticleКОМПЬЮТЕРНЫЕ НАУКИCOMPUTER SCIENCEРЕАЛИЗАЦИЯ МНОГОАГЕНТНОЙ АРХИТЕКТУРЫ IMPALA ДЛЯ УПРАВЛЕНИЯ ТЕМПЕРАТУРОЙ В ОДНОЙ ЗОНЕ СИМУЛИРОВАННОГО ЗДАНИЯIMPLEMENTATION OF MULTI-AGENT FRAMEWORK OF IMPALA FOR A SINGLE ZONE TEMPERATURE CONTROL OF A SIMULATED THERMAL ZONEhttps://orcid.org/0009-0004-4639-3548КаппароваА.KapparovaA.

Докторант.

Алматы

PhD student.

Almaty

kapparova_ainur3@kaznu.edu.kzhttps://orcid.org/0000-0001-8206-7425ЖоламановБ.ZholamanovB.

Докторант.

Алматы

PhD student.

Almaty

zholamanov.batyrbek@kaznu.kzhttps://orcid.org/0009-0004-7613-5507БолатбекА.BolatbekA.

Докторант.

Алматы

PhD student.

Almaty

bolatbek.askhat@kaznu.kzhttps://orcid.org/0000-0002-5723-6642КуттыбайН.KuttybayN.

PhD.

Алматы

PhD.

Almaty

kuttybyy.nurzhigit@kaznu.kzhttps://orcid.org/0000-0002-3935-7213ДосымбетоваГ.DosymbetovaG.

PhD.

Алматы

PhD.

Almaty

gulbakhar.dossymbetova@kaznu.edu.kzhttps://orcid.org/0009-0002-9213-2555ЖумагалиевЕ.ZhumagaliyevYe.

Магистрант.

Алматы

Master student.

Almaty

zhumagaliyev_y0@live.kaznu.kzКазахский национальный университет им. аль-ФарабиКазахстанAl-Farabi Kazakh National UniversityKazakhstan202627062026232218232Copyright © Каппарова А., Жоламанов Б., Болатбек А., Куттыбай Н., Досымбетова Г., Жумагалиев Е., 20262026Каппарова А., Жоламанов Б., Болатбек А., Куттыбай Н., Досымбетова Г., Жумагалиев Е.Kapparova A., Zholamanov B., Bolatbek A., Kuttybay N., Dosymbetova G., Zhumagaliyev Y.Данная работа распространяется под лицензией Creative Commons Attribution 4.0.This work is licensed under a Creative Commons Attribution 4.0 License.https://vestnik.kbtu.edu.kz/jour/article/view/2902

Здания составляют значительную долю мирового энергопотребления, в том числе системы отопления, вентиляции и кондиционирования воздуха (HVAC) и управления влажностью, формируют основную часть их эксплуатационных энергозатрат. Традиционные стратегии управления, основанные на фиксированных правилах, часто не способны адаптироваться к динамически изменяющимся внутренним и внешним условиям, что обуславливает необходимость применения методов управления на основе данных. В данном исследовании представлен фреймворк многоагентного обучения с подкреплением (MARL) для одновременного управления температурой и влажностью в однозонном здании, смоделированном в среде EnergyPlus. Предложенный подход использует распределенную архитектуру Importance Weighted Actor-Learner Architecture (IMPALA) с централизованным обучением и децентрализованным исполнением (CTDE), что позволяет двум агентам – температурному и влажностному – обучаться согласованным стратегиям непосредственно на основе обратной связи высокоточной симуляционной модели. Полученные результаты демонстрируют высокую эффективность обучения: оба агента существенно улучшили среднее вознаграждение на шаге (температура +18,9%, влажность +33,7%), что свидетельствует об успешной сходимости и кооперативном поведении. Обученный контроллер обеспечивает уровень теплового комфорта, сопоставимый с базовой стратегией управления на основе правил (средняя разница температуры в занятый период ≈ 0,04 °C; среднее значение PMV в занятый период ≈ 0,45), при одновременном достижении значительной экономии энергии. Совокупное годовое энергопотребление системы HVAC снижается на 8,9%, при этом наибольшее сокращение было достигнуто по энергии увлажнения на 34,4%. Нагрузки на отопление и охлаждение практически не изменяюся, что подтверждает достижение энергосбережения без ухудшения показателей комфорта.

Buildings account for a significant portion of global energy consumption, with HVAC and humidity-control systems representing the majority of their operational demand. Traditional rule-based strategies often fail to adapt to dynamic indoor-outdoor conditions, motivating the use of data-driven control methods. This study presents a multiagent reinforcement learning (MARL) framework for simultaneous temperature and humidity control in a singlezone building modeled in EnergyPlus. The proposed approach employs the distributed Importance Weighted ActorLearner Architecture (IMPALA) algorithm with centralized training and decentralized execution (CTDE), enabling two agents: temperature and humidity to learn coordinated policies directly from high-fidelity simulation feedback. The results demonstrate strong learning performance: both agents improved their per-step rewards substantially (temperature +18.9%, humidity +33.7%), indicating effective convergence and cooperative behavior. The learned controller maintained thermal comfort comparable to the rule-based baseline (mean occupied temperature difference ≈ 0.04 °C; occupied PMV ≈ 0.45) while achieving notable energy savings. Total annual HVAC energy consumption decreased by 8.9%, with the most significant improvement observed in humidification energy, which was reduced by 34.4%. Heating and cooling loads remained nearly unchanged, confirming that energy reductions were achieved without compromising comfort.

модель зданиямногоагентная архитектурамногоагентное обучение с подкреплениеминтеллектуальное управлениеImportance Weighted Actor-Learner Architecturebuilding modelmulti-agent frameworkmulti-agent reinforcement learningintelligent controlImportance Weighted Actor-Learner ArchitectureThis study was funded by the Committee of Science of the Ministry of Science and Higher Education of the Republic of Kazakhstan (grant No. AP26197509 Intelligent control system for building energy consumption and occupant comfort using multi-agent reinforcement learning)ReferencesDean, B., et al. Towards zero-emission efficient and resilient buildings. Global Status Report (2016).Dean, B., et al. Towards zero-emission efficient and resilient buildings. Global Status Report (2016).Razmara, M., et al. Optimal exergy control of building HVAC system. Applied Energy, 156, 555–565 (2015). https://doi.org/10.1016/j.apenergy.2015.07.051Razmara, M., et al. Optimal exergy control of building HVAC system. Applied Energy, 156, 555–565 (2015). https://doi.org/10.1016/j.apenergy.2015.07.051Chua, K.J., et al. Achieving better energy-efficient air conditioning – a review of technologies and strategies. Applied Energy, 104, 87–104 (2013). https://doi.org/10.1016/j.apenergy.2012.10.037Chua, K.J., et al. Achieving better energy-efficient air conditioning – a review of technologies and strategies. Applied Energy, 104, 87–104 (2013). https://doi.org/10.1016/j.apenergy.2012.10.037Maasoumy, M., et al. Handling model uncertainty in model predictive control for energy efficient buildings. Energy and Buildings, 77, 377–392 (2014). https://doi.org/10.1016/j.enbuild.2014.03.057Maasoumy, M., et al. Handling model uncertainty in model predictive control for energy efficient buildings. Energy and Buildings, 77, 377–392 (2014). https://doi.org/10.1016/j.enbuild.2014.03.057Salakij, S., et al. Model-Based Predictive Control for building energy management. I: Energy modeling and optimal control. Energy and Buildings, 133, 345–358 (2016). https://doi.org/10.1016/j.enbuild.2016.09.044Salakij, S., et al. Model-Based Predictive Control for building energy management. I: Energy modeling and optimal control. Energy and Buildings, 133, 345–358 (2016). https://doi.org/10.1016/j.enbuild.2016.09.044Yang, S., et al. Experiment study of machine-learning-based approximate model predictive control for energy-efficient building control. Applied Energy, 288, 116648 (2021). https://doi.org/10.1016/j.apenergy.2021.116648Yang, S., et al. Experiment study of machine-learning-based approximate model predictive control for energy-efficient building control. Applied Energy, 288, 116648 (2021). https://doi.org/10.1016/j.apenergy.2021.116648Crawley, D.B., et al. EnergyPlus: creating a new-generation building energy simulation program. Energy and Buildings, 33(4), 319–331 (2001). https://doi.org/10.1016/S0378-7788(00)00114-6Crawley, D.B., et al. EnergyPlus: creating a new-generation building energy simulation program. Energy and Buildings, 33(4), 319–331 (2001). https://doi.org/10.1016/S0378-7788(00)00114-6Strachan, P.A., Kokogiannakis, G., Macdonald, I.A. History and development of validation with the ESP-r simulation program. Building and Environment, 43(4), 601–609 (2008). https://doi.org/10.1016/j.buildenv.2006.06.025Strachan, P.A., Kokogiannakis, G., Macdonald, I.A. History and development of validation with the ESP-r simulation program. Building and Environment, 43(4), 601–609 (2008). https://doi.org/10.1016/j.buildenv.2006.06.025Salvalai, G. Implementation and validation of simplified heat pump model in IDA-ICE energy simulation environment. Energy and Buildings, 49, 132–141 (2012). https://doi.org/10.1016/j.enbuild.2012.01.038Salvalai, G. Implementation and validation of simplified heat pump model in IDA-ICE energy simulation environment. Energy and Buildings, 49, 132–141 (2012). https://doi.org/10.1016/j.enbuild.2012.01.038Shrivastava, R.L., Kumar, V., Untawale, S.P. Modeling and simulation of solar water heater: A TRNSYS perspective. Renewable and Sustainable Energy Reviews, 67, 126–143 (2017). https://doi.org/10.1016/j.rser.2016.09.005Shrivastava, R.L., Kumar, V., Untawale, S.P. Modeling and simulation of solar water heater: A TRNSYS perspective. Renewable and Sustainable Energy Reviews, 67, 126–143 (2017). https://doi.org/10.1016/j.rser.2016.09.005Koeln, J., et al. Multi-zone temperature modeling and control. In: Intelligent Building Control Systems: A Survey of Modern Building Control and Sensing Strategies. Springer, Cham, 139–166 (2017). https://doi.org/10.1007/978-3-319-68462-8_6Koeln, J., et al. Multi-zone temperature modeling and control. In: Intelligent Building Control Systems: A Survey of Modern Building Control and Sensing Strategies. Springer, Cham, 139–166 (2017). https://doi.org/10.1007/978-3-319-68462-8_6Perera, D.W.U., Pfeiffer, C.F., Skeie, N.-O. Control of temperature and energy consumption in buildings – a review. International Journal of Energy & Environment, 5(4) (2014).Perera, D.W.U., Pfeiffer, C.F., Skeie, N.-O. Control of temperature and energy consumption in buildings – a review. International Journal of Energy & Environment, 5(4) (2014).Sutton, R.S., Barto, A.G. Reinforcement Learning: An Introduction. MIT Press, Cambridge (1998). https://doi.org/10.1017/S0263574799271172Sutton, R.S., Barto, A.G. Reinforcement Learning: An Introduction. MIT Press, Cambridge (1998). https://doi.org/10.1017/S0263574799271172Li, F., Du, Y. Intelligent multi-zone residential HVAC control strategy based on deep reinforcement learning. In: Deep Learning for Power System Applications. Springer, Cham, 71–96 (2023). https://doi.org/10.1007/978-3-031-45357-1_4Li, F., Du, Y. Intelligent multi-zone residential HVAC control strategy based on deep reinforcement learning. In: Deep Learning for Power System Applications. Springer, Cham, 71–96 (2023). https://doi.org/10.1007/978-3-031-45357-1_4Barrett, E., Linder, S. Autonomous HVAC control, a reinforcement learning approach. In: ECML PKDD. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-23461-8_1Barrett, E., Linder, S. Autonomous HVAC control, a reinforcement learning approach. In: ECML PKDD. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-23461-8_1Mocanu, E., et al. On-line building energy optimization using deep reinforcement learning. IEEE Transactions on Smart Grid, 10(4), 3698–3708 (2018). https://doi.org/10.1109/TSG.2018.2834219Mocanu, E., et al. On-line building energy optimization using deep reinforcement learning. IEEE Transactions on Smart Grid, 10(4), 3698–3708 (2018). https://doi.org/10.1109/TSG.2018.2834219Li, Y., et al. Transforming cooling optimization for green data center via deep reinforcement learning. IEEE Transactions on Cybernetics, 50(5), 2002–2013 (2019). https://doi.org/10.1109/TCYB.2019.2927410Li, Y., et al. Transforming cooling optimization for green data center via deep reinforcement learning. IEEE Transactions on Cybernetics, 50(5), 2002–2013 (2019). https://doi.org/10.1109/TCYB.2019.2927410Liu, B., Akcakaya, M., McDermott, T.E. Automated control of transactive HVACs in energy distribution systems. IEEE Transactions on Smart Grid, 12(3), 2462–2471 (2020). https://doi.org/10.1109/TSG.2020.3042498Liu, B., Akcakaya, M., McDermott, T.E. Automated control of transactive HVACs in energy distribution systems. IEEE Transactions on Smart Grid, 12(3), 2462–2471 (2020). https://doi.org/10.1109/TSG.2020.3042498Kazmi, H., et al. Multi-agent reinforcement learning for modeling and control of thermostatically controlled loads. Applied Energy, 238, 1022–1035 (2019). https://doi.org/10.1016/j.apenergy.2019.01.140Kazmi, H., et al. Multi-agent reinforcement learning for modeling and control of thermostatically controlled loads. Applied Energy, 238, 1022–1035 (2019). https://doi.org/10.1016/j.apenergy.2019.01.140Blad, C., Bøgh, S., Kallesøe, C. A multi-agent reinforcement learning approach to price and comfort optimization in HVAC-systems. Energies, 14(22), 7491 (2021). https://doi.org/10.3390/en14227491Blad, C., Bøgh, S., Kallesøe, C. A multi-agent reinforcement learning approach to price and comfort optimization in HVAC-systems. Energies, 14(22), 7491 (2021). https://doi.org/10.3390/en14227491Espeholt, L., et al. IMPALA: Scalable distributed deep-RL with importance weighted actor-learner architectures. Proceedings of ICML (2018).Espeholt, L., et al. IMPALA: Scalable distributed deep-RL with importance weighted actor-learner architectures. Proceedings of ICML (2018).EnergyPlus Weather Data. URL: https://energyplus.net/weather (accessed: 2026.06.01).EnergyPlus Weather Data. URL: https://energyplus.net/weather (accessed: 2026.06.01).

The authors declare that there are no conflicts of interest present.