EARLY PREDICTION OF IN-HOSPITAL MORTALITY USING INTERPRETABLE AND CALIBRATED ENSEMBLE MACHINE LEARNING MODELS

Early identification of patients at high risk of in-hospital mortality is a persistent challenge in clinical practice, particularly during infectious disease outbreaks such as COVID-19, when decisions must be made at the time of hospital admission using incomplete information. Although machine learning methods have been widely applied to mortality prediction, many existing models are developed and assessed under conditions that limit their usefulness for early triage, including insufficient attention to probability reliability, temporal generalization, and decisionoriented assessment. This work examines early-stage mortality risk prediction using routinely available clinical data collected at the time of hospital admission. An ensemble of gradient boosting models is developed and analyzed under strict temporal separation to reflect real-world deployment conditions. Particular emphasis is placed on the reliability of predicted probabilities, with post-hoc isotonic calibration applied to improve alignment between predicted risks and observed outcomes. Model performance is assessed using complementary discrimination and calibration metrics, while clinical relevance is examined through decision curve analysis across plausible operating thresholds. Evaluation on an independent, temporally held-out test set shows that ensemble aggregation improves the stability of risk estimates, while calibration yields more consistent threshold-based behavior without altering ranking performance. Decision curve analysis indicates that calibrated predictions provide higher net benefit than default decision strategies across a broad range of early triage thresholds. These findings highlight that, in early clinical decision-making, probability reliability plays a critical role alongside discrimination. The presented framework offers a methodologically robust approach to early in-hospital mortality risk assessment under realistic clinical and temporal constraints.

1. Al-Nafjan, A., Aljuhani, A., Alshebel, A., Alharbi, A., Alshehri, A. Artificial intelligence in predictive healthcare: A systematic review. Journal of Clinical Medicine, 14 (19), 6752 (2025). https://doi.org/10.3390/jcm14196752

2. Bajwa, J., Munir, U., Nori, A., Williams, B. Artificial intelligence in healthcare: Transforming the practice of medicine. Future Healthcare Journal, 8 (2), e188–e194 (2021). https://doi.org/10.7861/fhj.2021-0095

3. Kelly, C. J., Karthikesalingam, A., Suleyman, M., Corrado, G., King, D. Key challenges for delivering clinical impact with artificial intelligence. BMC Medicine, 17, 195 (2019). https://doi.org/10.1186/s12916-019-1426-2

4. Wynants, L., Van Calster, B., Collins, G. S., Riley, R. D., Heinze, G., Schuit, E., COVID-19 Prediction Model Consortium. Prediction models for diagnosis and prognosis of COVID-19: Systematic review and critical appraisal. BMJ, 369, m1328 (2020). https://doi.org/10.1136/bmj.m1328

5. Rajwa, B., Naved, M. M. A., Adibuzzaman, M., Grama, A. Y., Khan, B. A., Dundar, M., Rochet, J.- C. Identification of predictive patient characteristics for assessing the probability of COVID-19 in-hospital mortality. PLOS Digital Health, 3 (4), e0000327 (2024). https://doi.org/10.1371/journal.pdig.0000327

6. Rudin, C., Chen, C., Chen, Z., Huang, H., Semenova, L., Zhong, C. Interpretable machine learning: Fundamental principles and 10 grand challenges. Statistics Surveys, 16, 1–85 (2022). https://doi.org/10.1214/21-SS133

7. Ennab, M., Mcheick, H. Enhancing interpretability and accuracy of AI models in healthcare: A comprehensive review on challenges and future directions. Frontiers in Robotics and AI, 11, 1444763 (2024). https://doi.org/10.3389/frobt.2024.1444763

8. Tonekaboni, S., Joshi, S., McCradden, M. D., Goldenberg, A. What clinicians want: Contextualizing explainable machine learning for clinical end use. Proceedings of Machine Learning Research, 106, 359–380 (2019). https://proceedings.mlr.press/v106/tonekaboni19a.html

9. Van Calster, B., McLernon, D. J., van Smeden, M., Wynants, L., Steyerberg, E. W. Calibration: The Achilles heel of predictive analytics. BMC Medicine, 17, 230 (2019). https://doi.org/10.1186/s12916-019-1466-7

10. Vickers, A. J., Van Calster, B., Steyerberg, E. W. Net benefit approaches to the evaluation of prediction models, molecular markers, and diagnostic tests. BMJ, 352, i6 (2016). https://doi.org/10.1136/bmj.i6

11. Sadatsafavi, M., Adibi, A., Puhan, M., Gershon, A., Aaron, S. D., Sin, D. D. Moving beyond AUC: Decision curve analysis for quantifying net benefit of risk prediction models. European Respiratory Journal, 58 (5), 2101186 (2021). https://doi.org/10.1183/13993003.01186-2021

12. Andaur Navarro, C. L., Damen, J. A. A., van Smeden, M., Takada, T., Nijman, S. W. J., Dhiman, P., Ma, J., Collins, G. S., Bajpai, R., Riley, R. D., Moons, K. G. M., Hooft, L. Systematic review identifies the design and methodological conduct of studies on machine learning-based prediction models. Journal of Clinical Epidemiology, 154, 1–12 (2023). https://doi.org/10.1016/j.jclinepi.2022.11.015

13. Giddings, R., Joseph, A., Callender, T., Janes, S. M., van der Schaar, M., Sheringham, J., Navani, N. Factors influencing clinician and patient interaction with machine learning-based risk prediction models: A systematic review. The Lancet Digital Health, 6 (2), e131–e144 (2024). https://doi.org/10.1016/S2589-7500(23)00241-8

14. Subbaswamy, A., Saria, S. From development to deployment: Dataset shift, causality, and shift-stable models in health AI. Biostatistics, 21 (2), 345–352 (2020). https://doi.org/10.1093/biostatistics/kxz041

15. Lu, J., Liu, A., Dong, F., Gu, F., Gama, J., Zhang, G. Learning under concept drift: A review. IEEE Transactions on Knowledge and Data Engineering, 31 (12), 2346–2363 (2019). https://doi.org/10.1109/TKDE.2018.2876857

16. dos Santos Silva, G. F., Barcellos Filho, F. N., Wichmann, R. M., da Silva Junior, F. C., Chiavegatto Filho, A. D. P. Strategies for detecting and mitigating dataset shift in machine learning for health predictions: A systematic review. Journal of Biomedical Informatics, 170, 104902 (2025). https://doi.org/10.1016/j.jbi.2025.104902

17. Riley, R. D., Archer, L., Snell, K. I. E., Ensor, J., Dhiman, P., Martin, G. P., Bonnett, L. J., Collins, G. S. Evaluation of clinical prediction models (part 2): How to undertake an external validation study. BMJ, 384, e074820 (2024). https://doi.org/10.1136/bmj-2023-074820

18. Austin, P. C., Steyerberg, E. W. The Integrated Calibration Index (ICI) and related metrics for quantifying the calibration of logistic regression models. Statistics in Medicine, 38 (21), 4051–4065 (2019). https://doi.org/10.1002/sim.8281

19. Ojeda, F. M., Jansen, M. L., Thiéry, A., Blankenberg, S., Weimar, C., Schmid, M., Ziegler, A. Calibrating machine learning approaches for probability estimation: A comprehensive comparison. Statistics in Medicine, 42 (29), 5451–5478 (2023). https://doi.org/10.1002/sim.9921

20. Gneiting, T., Raftery, A. E. Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102 (477), 359–378 (2007). https://doi.org/10.1198/016214506000001437

21. Chalkou, K., Vickers, A. J., Pellegrini, F., Manca, A., Salanti, G. Decision curve analysis for personalized treatment choice between multiple options. Medical Decision Making, 43 (3), 337–349 (2022). https://doi.org/10.1177/0272989X221143058

22. Piovani, D., Sokou, R., Tsantes, A. G., Vitello, A. S., Bonovas, S. Optimizing clinical decision making with decision curve analysis: Insights for clinical investigators. Healthcare, 11 (16), 2244 (2023). https://doi.org/10.3390/healthcare11162244