Advertisement

External Validation of a Laboratory Prediction Algorithm for the Reduction of Unnecessary Labs in the Critical Care Setting

Published:January 30, 2022DOI:https://doi.org/10.1016/j.amjmed.2021.12.020

      Abstract

      Background

      Unnecessary laboratory tests contribute to iatrogenic harm and are a major source of waste in the health care system. We previously developed a machine learning algorithm to help clinicians identify unnecessary laboratory tests, but it has not been externally validated. In this study, we externally validate our machine learning algorithm.

      Methods

      To externally validate the machine learning algorithm that was originally trained on the Medical Information Mart for Intensive Care (MIMIC) III database, we tested the algorithm in a separate institution. We identified and abstracted data for all patients older than 18 years admitted to the intensive care unit at Memorial Hermann Hospital in Houston, Texas (MHH) from January 1, 2020 to November 13, 2020. Using the transfer learning style, we performed external validation of the machine learning algorithm.

      Results

      A total of 651 MHH patients were included. The model performed well in predicting abnormality (area under the curve [AUC] 0.98 for MIMIC III and 0.89 for MHH). The model performed similarly in predicting transitions from normal laboratory range to abnormal (AUC 0.71 for MIMIC III and 0.70 for MHH). The performance of the model in predicting the actual laboratory value was also similar in the MIMIC III (accuracy 0.41) and MHH data (0.45).

      Conclusions

      We externally validated the machine learning model and showed that the model performed similarly, supporting the generalizability to other settings. While this model demonstrated good performance for predicting abnormal labs and transitions, it does not perform well enough for prediction of laboratory values in most clinical applications.

      Keywords

      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'

      Subscribe:

      Subscribe to The American Journal of Medicine
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect

      References

        • Shrank WH
        • Rogstad TL
        • Parekh N
        Waste in the US health care system: estimated costs and potential for savings.
        JAMA. 2019; 322 (Available at) (Accessed July 24, 2020): 1501-1509
        • Zhi M
        • Ding EL
        • Theisen-Toupal J
        The landscape of inappropriate laboratory testing: a 15-year meta-analysis.
        PLoS One. 2013; 8 (Available at:) (Accessed January 28, 2021): e78962
      1. Janowiak D, Hannon T. The laboratory's role in delivering high-value care. 2018. Available at: https://clpmag.com/diagnostic-technologies/hematology-serology/laboratorys-role-delivering-high-value-care/. Accessed September 30, 2021.

        • Chornenki NLJ
        • James TE
        • Barty R
        • et al.
        Blood loss from laboratory testing, anemia, and red blood cell transfusion in the intensive care unit: a retrospective study.
        Transfusion (Paris). 2020; 60 (Available at:) (Accessed January 7, 2021): 256-261
        • Corwin HL
        • Parsonnet KC
        • Gettinger A
        RBC transfusion in the ICU: is there a reason?.
        Chest. 1995; 108 (Available at: http://www.sciencedirect.com/science/article/pii/S0012369216342295. Accessed January 7, 2021): 767-771
        • Thavendiranathan P
        • Bagai A
        • Ebidia A
        Do blood tests cause anemia in hospitalized patients? The effect of diagnostic phlebotomy on hemoglobin and hematocrit levels.
        J Gen Intern Med. 2005; 20 (Available at:) (Accessed July 24, 2020): 520-524
        • Lacroix J
        • Boven K
        • Forbes P
        Anemia, blood loss, and blood transfusions in North American children in the intensive care unit.
        Am J Respir Crit Care Med. 2008; 178 (Available at:) (Accessed July 24, 2020): 26-33
        • Koch CG
        • Li L
        • Sun Z
        Hospital-acquired anemia: Prevalence, outcomes, and healthcare implications.
        J Hosp Med. 2013; 8 (Available at:) (Accessed July 24, 2020): 506-512
        • Makam AN
        • Nguyen OK
        • Clark C
        Incidence, predictors, and outcomes of hospital-acquired anemia.
        J Hosp Med. 2017; 12 (Available at:) (Accessed January 7, 2021): 317-322
        • Salisbury AC
        • Amin AP
        • Reid KJ
        Hospital-acquired anemia and in-hospital mortality in patients with acute myocardial infarction.
        Am Heart J. 2011; 162: 300-309.e3
      2. Choosing Wisely: An Initiative of the ABIM Foundation. American Association of Blood Banks - serial blood counts. Available at: https://www.choosingwisely.org/clinician-lists/american-association-blood-banks-serial-blood-counts-on-clinically-stable-patients/. Accessed July 24, 2020.

      3. Choosing Wisely: An Initiative of the ABIM Foundation. Critical Care Societies Collaborative – Critical Care: responsive diagnostic tests. Available at: https://www.choosingwisely.org/clinician-lists/critical-care-societies-collaborative-regular-diagnostic-tests/. Accessed July 24, 2020.

        • Bindraban RS
        • ten Berg MJ
        • Naaktgeboren CA
        • Kramer MHH
        • Van Solinge WW
        • Nanayakkara PWB
        Reducing test utilization in hospital settings: a narrative review.
        Ann Lab Med. 2018; 38 (Available at:) (Accessed January 7, 2021): 402-412
        • Ekblom K
        • Petersson A
        Introduction of cost display reduces laboratory test utilization.
        Am J Manag Care. 2018; 24: e164-e169
        • Faisal A
        • Andres K
        • Rind JAK
        • et al.
        Reducing the number of unnecessary routine laboratory tests through education of internal medicine residents.
        Postgrad Med J. 2018; 94 (Available at:) (Accessed July 24, 2020): 716-719
        • May TA
        • Clancy M
        • Critchfield J
        Reducing unnecessary inpatient laboratory testing in a teaching hospital.
        Am J Clin Pathol. 2006; 126 (Available at:) (Accessed July 24, 2020): 200-206
        • Cismondi F
        • Celi LA
        • Fialho AS
        • et al.
        Reducing unnecessary lab testing in the ICU with artificial intelligence.
        Int J Med Inf. 2013; 82 (Available at:) (Accessed December 21, 2020): 345-358
        • Aikens RC
        • Balasubramanian S
        • Chen JH
        A machine learning approach to predicting the stability of inpatient lab test results.
        AMIA Jt Summits Transl Sci Proc. 2019; (Available at:) (Accessed September 18, 2019): 515-523
        • Luo Y
        • Szolovits P
        • Dighe AS
        • Baron JM
        Using machine learning to predict laboratory test results.
        Am J Clin Pathol. 2016; 145 (Available at:) (Accessed September 1, 2020): 778-788
        • Yu L
        • Li L
        • Bernstam E
        A deep learning solution to recommend laboratory reduction strategies in ICU.
        Int J Med Inf. 2020; (144. Available at:) (Accessed October 8, 2020)104282
        • Johnson AEW
        • Pollard TJ
        • Shen L
        • et al.
        MIMIC-III, a freely accessible critical care database.
        Sci Data. 2016; 3 (Available at:) (Accessed September 18, 2019)160035
        • Yu L
        • Zhang Q
        • Bernstam EV
        • Jiang X
        Predict or draw blood: an integrated method to reduce lab tests.
        J Biomed Inform. 2020; 104 (Available at:) (Accessed June 1, 2020)103394
      4. Gupta P, Malhotra P, Narwariya J, Vig L, Shroff G. Transfer learning for clinical time series analysis using deep neural networks. arXiv:190400655. 2021. Available at: http://arxiv.org/abs/1904.00655. Accessed October 14, 2021.

      5. Mokrii I, Boytsov L, Braslavski P. A systematic evaluation of transfer learning and pseudo-labeling with BERT-based ranking models. arXiv:2013.03335. 2021. Available at: http://arxiv.org/abs/2103.03335. Accessed October 14, 2021.

        • Heyen NB
        • Salloch S
        The ethics of machine learning-based clinical decision support: an analysis through the lens of professionalisation theory.
        BMC Med Ethics. 2021; 22 (Available at: Accessed November 18, 2021): 112https://doi.org/10.1186/s12910-021-00679-3