CAREER: Knowledge-enhanced and interpretable radiology report generation

Joan And Sanford I. Weill Medical College Of Cornell University, New York NY

Investigators

Abstract

The radiology report is the primary mean of communication between radiologists and referring physicians, and which also serve as a legal document. To date, many studies have demonstrated the feasibility of using deep learning to automatically generate radiology reports from chest x-rays. However, existing approaches utilize only current chest x-ray images and do not consider historical images, associated electronic health records (EHRs), and domain-specific prior knowledge. Therefore, the current computer-generated reports are far from accurate and complete. To bridge this gap, there is a critical need to study new report generation techniques to handle large-scale, real-world healthcare data. This project will employ novel informatics and data science techniques to automatically generate clinical reports to improve workflow efficiency and improve healthcare outcomes. From the perspectives of biomedical informatics, our approach will leverage the wealth of information from EHR to profoundly understand the role of natural language, image analysis, and deep learning in report generation. From the perspective of clinical translation, this project will facilitate radiologists’ workflow, improve clinical accuracy and efficiency, and enhance decision-making. Additionally, the project will closely integrate research with education, by launching a new graduate Natural Language Processing and Health course and supporting several capstone and specialization projects. It will also broaden the outreach from the investigators to non-computer-science graduate students, who will be exposed to working principles of NLP through our extensive collaborative efforts. This project will develop and validate a framework to automatically generate radiology reports using longitudinal, multimodal EHR data and domain knowledge. The investigator will attain the overall objective by pursuing four aims. First, the project will build a memory-enhanced report generation system to handle longitudinal chest x-rays and reports. Second, the project will build a radiology-specific knowledge graph from multimodal EHR and inject it into the report generation framework. We will employ a novel approach to construct such radiology-specific knowledge graph, by modeling heterogeneous multi-dimensional EHR data in our model. Third, we will create a new rationale-based model that supports rationale-base interpretabilityFinally, the project will build and evaluate a prototype user-centered reporting system with a user-friendly graphic user interface. The new reporting system will enhance communication between radiologists and referral physicians, particularly in large and heterogeneous EHR. The proposed research is creative and original because it represents a step towards building automatic systems with a higher-level understanding of radiology knowledge and decision-making. It is expected to open research horizons and employ techniques and theories from data science to support next-generation medical diagnostic reasoning from chest x-rays and structured EHR. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.