The impact of artificial intelligence-driven risk prediction on clinical decisions in heart failure patients: the design of an international vignette study

Heart failure (HF), affecting over 64 million people worldwide, is a complex syndrome that requires personalized treatment. [1] The use of risk prediction models may help guide tailored care and is recommended by international guidelines. [2] However, clinical utility and model impact on clinical decision-making in HF care remain largely uninvestigated. [3,4] The AI4HF consortium (Trustworthy Artificial intelligence for Personalized Risk Assessment in Chronic Heart Failure) is developing HF risk prediction models for use at the ED, at hospital discharge and for decisions regarding referral to specialized advanced HF teams. These models, with their outcomes and explainability shown in a web-based interface, will offer a valuable opportunity to evaluate the impact of AI-driven risk predictions on HF decision making.

We present the design of a clinical vignette study to evaluate the impact of the use of predictive models on the clinical decision making in three different hospital settings.

Questionnaires distributed in preparation for the vignette study (filled in by 24 cardiologists and residents from the eight participating hospitals) revealed that clinicians anticipate risk-scores to be helpful in the three clinical settings (see figure 1). Therefore, we consider it valuable to evaluate the models in all three settings. During the study, eighty clinicians from eight hospitals will evaluate clinical vignettes based on real-world data extracted from electronic health records (EHR) from patients presenting at the ED, hospitalized for acute HF, or visiting the outpatient clinic with non-ischemic dilated cardiomyopathy. For the proposed study, 375 vignettes will be created and clinicians will be asked to take clinical decisions and indicate their decision certainty and trust in the risk prediction model. Multiple AI4HF models are available for each clinical setting, that differ in their predicted outcome or training population. Each clinician will assess 25 vignettes, first without access to a model, and subsequently reassess the same vignette with one of the models present alongside the vignette. Finally, clinicians reassess the same vignette while they have access to follow-up data from the subsequent period (up to five years if available) serving as a reference standard. As an outcome, we will assess whether the use of a model leads to changes in clinical decisions, and study associations between those changes and clinician-, vignette- and model-related factors. Besides, we will use binomial tests to evaluate whether the use of each model improves agreement with the prospective decisions.

This study will provide insight into the impact of an AI-based risk prediction tool on clinical decision making of HF patients. Together with patient focus group results, outcomes of this study will guide implementation of AI for heart failure.