Usability of novel beat-by-beat ECG annotation software portal for generating large-scale machine learning training datasets

Despite the availability of numerous large publicly available ECG datasets, there is a distinct absence of specialized software to enable the beat-by-beat annotation and labeling, limiting the ability to generate high-quality machine learning training datasets. This study analyzed the usability of such a novel software interface. Through a survey of several cardiologists and clinic staff, several metrics were gauged, including user satisfaction, system and app simplicity, user interface, and effectiveness. The survey results gleaned insight into the users’ experiences, providing an understanding of the software’s user-friendliness and potential in developing a large-scale machine learning annotated ECG training dataset to advance cardiac health research.

3 cardiology physicians, and 9 cardiac telemetry unit staff were surveyed following use of the SafeBeat Annotation Software Platform to annotate a large publicly available dataset ECGs (n=10,646) beat-by-beat. The validated Post-Study System Usability Questionnaire (PSSUQ) and mobile Health App Usability Questionnaire (MAUQ) were used to assess the overall usability of the software interface. Results are reported as mean ± standard deviation.

Users completed 10,646 ECG beat-by-beat annotations in 62 days, and averaged 164.0 ECGs per day. Physicians rated the software interface highly, with an overall mean PSSUQ of 6.4 ± 0.8 and MAUQ of 6.4 ± 0.7. Staff also rated the software interface highly, with an overall mean PSSUQ of 6.7 ± 0.6 and MAUQ of 6.8 ± 0.5. Overall 92% of the users reported that they strongly agreed they would use the application again and 83% strongly agreed that it was easy to learn how to use the system.

The usability analysis of this novel beat-by-beat ECG annotation software revealed that it was a highly effective tool for both cardiologists and clinic staff. Both the system and app were reported to be simple to learn, comfortable to use, and easily navigable. With most users responding that they could become productive quickly using this system, there is clear potential for it to be used to develop a large-scale machine learning dataset.Figure 1