Qualitative evaluation of clinician interaction with a machine learning algorithm for the assessment of patients with suspected acute heart failure in the emergency department

N-terminal pro-B-type natriuretic peptide (NT-proBNP) assays have not been consistently implemented in practice despite being recommended in clinical guidelines for the assessment of acute heart failure. CODE-HF is a clinical decision-support tool that applies machine learning and NT-proBNP as a continuous measure and selected simple objective clinical variables to improve the diagnostic performance of NT-proBNP for acute heart failure.

In a qualitative study, we aimed to explore the acceptance, and barriers and facilitators that led to positive clinician engagement with CODE-HF when used to assess anonymised clinical cases.

Individual semi-structured interviews were conducted either face-to-face or by video call with 17 clinicians from different disciplines working in Emergency Departments at 3 hospitals. They were asked to review five anonymised clinical cases and ‘think aloud’ about how they would assess the patient, and their interpretation of the CODE-HF metrics. These include a score of 0-100 representing an individualised probability of acute heart failure, diagnostic metrics and a classification of low, intermediate or high probability of acute heart failure (Figure 1). Interviews were audio recorded, transcribed and coded. Codes were mapped onto the four domains of the Unified Theory of Acceptance and Use of Technology model (performance expectancy, effort expectancy, social influences, facilitating conditions).

Performance expectancy: Assessment could be improved using CODE-HF by facilitating objective communication between colleagues in a similar away to other widely used tools. The classification by probability score helped to reprioritise acute heart failure in cases where a diagnosis may have been missed. Effort expectancy: Statements relating to the positive or negative predictive value of a diagnosis of acute heart failure were viewed as useful information along with a visual traffic light system for the low-, intermediate- or high-probability categories. The absolute score was considered less useful to clinicians due to increased effort required for interpretation. Social influences: local and national guidelines carried the greatest weight of whether clinical decision support tools are used in practice, though respected research active colleagues and review on professional podcasts were also influential. Facilitating conditions: Access to a computer and clinical sample processing time were the only potential organisational issues identified as barriers. Clinicians were unanimous that clinical decision support tools provide supplementary information rather than replace clinical assessment which is central to the decision making process.

Clinicians reported that CODE-HF was a useful tool in the assessment of patients with breathlessness in the Emergency Department and identified the diagnostic metrics that were most helpful in guiding clinical decisions.

CODE-HF display