Research Initiation Award: Integrating Image and Text Information for Biomedical Literature-Based Cross and Multimodal Retrieval

Morgan State University, Baltimore MD

Investigators

Abstract

The Historically Black Colleges and Universities-Undergraduate Program (HBCU-UP) Research Initiation Awards (RIAs) provide support to STEM junior faculty at HBCUs who are starting to build a research program, as well as for mid-career faculty who may have returned to the faculty ranks after holding an administrative post or who needs to redirect and rebuild a research program. Faculty members may pursue research at their home institution, at an NSF-funded Center, at a research intensive institution or at a national laboratory. The RIA projects are expected to help further the faculty member's research capability and effectiveness, to improve research and teaching at his or her home institution, and to involve undergraduate students in research experiences. With support from the National Science Foundation, Morgan State University will conduct research in information retrieval using search strategies based on techniques from image processing as well as natural language processing. This would enable public access to both visual information and take away messages from journal articles. This project will provide valuable research experience and mentorship for several minority undergraduate students at Morgan State University. In addition, the project will help Morgan State University build its research capacity and enhance the educational and research experiences of their undergraduate students. Within the larger goal of expanding queries for information retrieval, the project will 1) use a crowdsourcing based approach to perform large scale manual annotation of visual regions of interest (ROIs) by pairing automatically detected ROIs to concepts occurring in a brief caption, 2) use a feature learning approach to extract discriminative features from ROIs and automatically map the ROIs to concepts in an existing textual ontology, such as RadLex, 3) aided by a visual ontology, consider the semantic relations between the visual words when assessing the distance between images described with the bag-of-visual-words feature representation scheme, 4) in addition to cross modal search by mapping image regions to concepts in ontology, perform multimodal search by fusing weighted text and image features generated by a multi-response linear regression (MLR)-based meta-learner in a classification-driven task-specific manner, and 5) evaluate the retrieval techniques using benchmark and realistic datasets by participating in the yearly ImageCLEF retrieval evaluation campaign. The labeled set of biomedical images with annotated regions of interest will be made available to the research community.