DigiLearnHF - an LLM-enhanced digital learning program for patients with implantable defibrillators and heart failure

Patient education is essential for patients with chronic heart failure and implantable cardioverter-defibrillator (ICD), yet public tools often rely on anecdotal or unverified sources, risking misinformation. The aim of our research project was to develop a structured digital learning program based solely on data verified by cardiac societies, using a Large Language Model (LLM) to aid content creation. Platform feasibility was subsequently assessed.

For the development of DigiLearnHF we 1) collected content from authoritative sources, including clinical guidelines and society-endorsed websites using the following keywords: guideline; heart failure; sudden cardiac death; cardiovascular implantable electronic devices. 2) To select a generative language model, we performed a structured comparison of four LLM-based chatbots with PDF-upload capabilities as of September 2024. Evaluation criteria included technical features (file upload, image generation, context window/token limit) and content quality (accuracy and scope). 3) Educational content (texts, tables, PFDs, video scripts, images) was generated using prompt engineering techniques and integrated into a digital learning platform.

For the feasibility study, patients with heart failure and implantable defibrillator used the platform for four weeks. User interaction was tracked digitally. Feasibility was assessed using the System Usability Score (SUS), measuring the perceived usability, the user version of the Mobile Application Rating Scale (uMARS), measuring the platform’s quality from end user’s perspective, and module-specific questionnaires rating the content.

Based on the comparison of the four LLMs, ChatGPT (GPT-4o, OpenAI) was identified as the most suitable LLM for developing DigiLearnHF. For detailed results regarding the LLM analysis see Figure 1. During development, hallucinations remained a major issue that could only partially be solved through prompt refinement and required expert control. For the detailed development process see Figure 2.

In the feasibility study, 25 patients were included (88% male, mean age 66.7 ± 15.7 years). Each patient used the platform for an average of 97.9 ± 52.0 minutes. A total of 17/25 (68%) participants completed all modules. The platform achieved a mean SUS score of 80.9/100 ± 14.8, indicating above-average usability. The uMARS scores showed a mean platform quality of 4.2/5 ± 0.5 points, a subjective quality of 3.3/5 ± 0.8 points, and a perceived impact score of 4.2/5 ± 0.7 points.

Structured development of a digital learning platform with LLM-assisted content creation is best achieved using the LLM ChatGPT. Ensuring content accuracy requires close expert oversight throughout the development process. Educating patients with heart failure and ICD using a digital learning platform like DigiLearnHF is feasible. However, adherence varied, suggesting the need for more tailored patient selection.Figure 1:

Content and technical analysis