Machine learning for medical imaging: automated disease diagnosis and prognosis

National Library Of Medicine

Investigators

Linked publications, trials & patents

Abstract

Deep learning, a class of machine learning algorithms, has showed impressive results in several of our recent studies this year. In addition to its application to natural language processing, we have also seen its success in our medical image analysis such as processing chest X-rays, CT images, and various kinds of retinal images for autonomous disease diagnosis and prognosis. Together with our clinical collaborators on the NIH campus, we have previously text-mined over 100,000 radiology reports where our algorithm generated weak training labels to enable the development of advanced deep learning methods for automatically reading and classifying chest X-ray images. This work resulted in the release to the scientific community of ChestX-ray14 (https://nihcc.app.box.com/v/ChestXray-NIHCC), one of the largest publicly available chest x-ray datasets with over 10,000 downloads. We have also conducted research to assist in the screening of age-related macular degeneration (AMD), a leading cause of vision loss in Americans 60 and older. By leveraging cutting-edge deep learning techniques and repurposing big imaging data from a major AMD clinical trial, we previously developed a novel data-driven approach (DeepSeeNet) for autonomous AMD diagnosis that exceeded the performance of human ophthalmologists (retinal specialists in this case). Such a result highlights the potential of deep learning systems to assist in early disease detection and enhance the clinical decision-making processes. In 2023, we continued this line of research with an emphasis on other diseases such as a specific form of late AMD called geographic atrophy (GA) and new imaging modalities in radiology such as optical coherence tomograph (OCT) images. Geographic atrophy (GA) is the defining lesion of the atrophic form of late AMD. It is predicted that, by 2040, atrophic AMD will affect > 5 million people worldwide. The detection of GA has important implications in both clinical practice and research. The diagnosis and severity classification of GA typically require in-person evaluation by an ophthalmologist. This may limit the accessibility of GA assessment, particularly for individuals living in remote areas or in countries with few ophthalmologists. In this context, automated approaches to GA diagnosis and classification have important potential advantages, including speed and accessibility, as well as accuracy and consistency. With Dr. Chew and her colleagues at NEI, we used 1284 SD-OCT scans from 311 participants to develop Deep-GA-Net: a 3-dimentaional (3d) deep learning network with 3D attention layer, for the detection of GA on spectral domain OCT (SD-OCT) scans. Cross-validation was used to evaluate Deep-GA-Net, where each testing set contained no participant from the corresponding training set. En face heatmaps and important regions at the B-scan level were used to visualize the outputs of Deep-GA-Net, and 3 ophthalmologists graded the presence or absence of GA in them to assess the explainability (i.e., understandability and interpretability) of its detections. Compared with other networks, Deep-GA-Net achieved the best metrics, with accuracy of 0.93, AUC of 0.94, and APR of 0.91, and received the best gradings of 0.98 and 0.68 on the en face heatmap and B-scan grading tasks, respectively. In summary, Deep-GA-Net was able to detect GA accurately from SD-OCT scans. The visualizations of Deep-GA-Net were more explainable, as suggested by 3 ophthalmologists. In addition to automatic disease diagnosis, our deep learning approach can significantly contribute to clinical research. For example, our previous DeepSeeNet software played a critical role in the recent clinical investigation to determine whether reticular pseudodrusen (RPD) status, ARMS2/HTRA1 genotype, or both are associated with altered geographic atrophy (GA) enlargement rate and to analyze potential mediation of genetic effects by RPD status. In radiology, we focused on automatically pre-filling radiology reports, an important clinical task that remains challenging despite various attempts in the past. Together with Dr. Summers and his team at the NIH clinical center, we proposed to use longitudinal multi-modal data, i.e., previous patient visit CXR, current visit CXR, and previous visit report, to pre-fill the findings section of a current patient visit report. We first gathered the longitudinal visit information for 26,625 patients from the MIMIC-CXR dataset and created a new dataset called Longitudinal-MIMIC. With this new dataset, a transformer-based model was trained to capture the in- formation from longitudinal patient visit records containing multi-modal data (CXR images + reports) via a cross-attention-based multi-modal fusion module and a hierarchical memory-driven decoder. In contrast to previous work that only uses current visit data as input to train a model, our work exploits the longitudinal information available to pre-fill the findings section of radiology reports. Experiments show that our approach outperforms several recent approaches significantly on F1 score, BLEU-4, METEOR and ROUGE-L respectively.