The quality of reporting in cardiac MRI artificial intelligence segmentation studies - a systematic review

Type of funding sources: Public Institution(s). Main funding source(s): This work was supported by an NIHR AI Award, AI_AWARD01706. This research was also funded in part, by the Wellcome Trust [Grant number 205188/Z/16/Z ].

There has been a rapid increase in the number of Artificial Intelligence (AI) studies of cardiac MRI (CMR) segmentation. AI has huge potential to improve image analysis assessments. However, advancement and clinical translation in this field depend on researchers presenting their work in a transparent and reproducible manner.

This systematic review aimed to evaluate the quality of reporting in AI studies involving CMR segmentation.

MEDLINE and EMBASE databases were searched for AI CMR segmentation studies on 18/11/2021. The flow of study inclusion is shown in Figure 1. Any AI method to segment any cardiac structure on CMR was eligible for inclusion. Each study was assessed for compliance with the Checklist for Artificial Intelligence in Medical Imaging (CLAIM).

70 studies were included in the qualitative analysis. Studies were published between 2015 to 2021, with the majority (71%) published in 2020 and 2021. Most studies were performed in Europe (33%), China (27%) and the USA (26%). Short-axis sections were segmented in 70% of studies and most commonly included both ventricles (51%) or the left ventricle alone (30%). 20 different architecture implementations were represented. Figure 2 summarises the most relevant CLAIM domains to AI segmentation. The training sample eligibility criteria, demographics and clinical characteristics were not reported in 47% and 81% of studies, respectively. Ground truth annotations, source of the annotations and annotation tool were absent in 31%, 36% and 51% of studies respectively. Preprocessing steps and software libraries and packages used in training were not included in 27% and 24%. Details on the training approach including the number of models trained and method of selecting the final model were missing in 20% and 17% of the studies. Methods of validation or testing on external data, inter- and intra- rater variability and failure analysis were unreported in 57%, 63% and 74%, respectively.

This systematic review highlights important gaps in the AI literature of CMR studies. We identified key items missing in the dataset description, model development, validation and testing that limit the transparency, reproducibility and hence validity of published AI studies. This review may support closer adherence to established frameworks for reporting standards.