III: Small: Reliable and Generalizable Neural Search Engine Architectures

Carnegie Mellon University, Pittsburgh PA

Investigators

Abstract

Scientists need to frequently search the scientific literature on the subject they are studying. Despite the availability of papers and citation databases on the Web, the enormous growth of scientific publications in all disciplines makes this a daunting task. Traditional commerical search engines, such as Google, often fail to include the most important documents in the first few pages of returned results - in other words, they do not do a good enough job of ranking scientific papers for a given query. Recently, new algorithms for search based on artificial neural network techniques have emerged as an alternative to traditional search architectures. These new neural search architectures are more accurate, but must be first trained with millions of example queries and answers from user interactions; this limits their usefulness for many tasks. This project will overcome this problem by developing new methods of training neural search engines that reduce the need for training examples by integrating explicit knowledge resources for a given discipline. The new techniques will be disseminated in freely available open-source search software for both university and industry researchers, thus broadly benefiting scientific advancement. In addition, the project will broaden participation by under-represented groups by creating research opportunities for female and undergraduate students and technology transfer opportunities for industry. This research develops new methods of training neural ranking architectures when a massive amount of training data is not available for the target application; integrates external knowledge resources to provide more information for making accurate ranking decisions; and applies the architecture to a domain-specific search task such as retrieving tabular data from scientific documents. This collection of problems is chosen to increase the practicality of neural ranking architectures outside of high-traffic commercial search environments, and to investigate and exploit the strengths of neural ranking architectures at using attention mechanisms to manage evidence, soft-matching across different types of evidence, and learning sophisticated nonlinear decision models. This research furthers the development of neural ranking architectures that are generally applicable and more reliable than current systems due to their ability to integrate a broader range of evidence in a predictable manner. Neural ranking architectures have generated much excitement and skepticism during the last several years. This research extends a recently-developed neural ranking system that is already able to beat strong learning-to-rank systems under specific conditions. It addresses one of the main obstacles to wider use of these models -- the availability of large amounts of training data. It integrates information from external semi-structured knowledge resources, because such information is effective in other ranking architectures and because it is likely to benefit from how neural ranking architectures manage and use diverse evidence of varying quality. Finally, it stress tests the architecture by applying it to a domain-specific task such as table retrieval from scientific documents, that requires the search engine to use several parts of the document selectively, rather than the entire document. These activities are designed to produce a neural ranking architecture capable of managing diverse evidence and document structure so as to provide greater knowledge about the particular strengths and weaknesses of neural ranking architectures. This research develops new methods of training neural ranking architectures when a massive amount of training data is not available for the target application; integrates external knowledge resources to provide more information for making accurate ranking decisions; and applies the architecture to a domain-specific search task such as retrieving tabular data from scientific documents. This collection of problems is chosen to increase the practicality of neural ranking architectures outside of high-traffic commercial search environments, and to investigate and exploit the strengths of neural ranking architectures at using attention mechanisms to manage evidence, soft-matching across different types of evidence, and learning sophisticated nonlinear decision models. This research furthers the development of neural ranking architectures that are generally applicable and more reliable than current systems due to their ability to integrate a broader range of evidence in a predictable manner. Neural ranking architectures have generated much excitement and skepticism during the last several years. This research extends a recently-developed neural ranking system that is already able to beat strong learning-to-rank systems under specific conditions. It addresses one of the main obstacles to wider use of these models -- the availability of large amounts of training data. It integrates information from external semi-structured knowledge resources, because such information is effective in other ranking architectures and because it is likely to benefit from how neural ranking architectures manage and use diverse evidence of varying quality. Finally, it stress tests the architecture by applying it to a domain-specific task such as table retrieval from scientific documents, that requires the search engine to use several parts of the document selectively, rather than the entire document. These activities are designed to produce a neural ranking architecture capable of managing diverse evidence and document structure so as to provide greater knowledge about the particular strengths and weaknesses of neural ranking architectures. The project website (http://www.cs.cmu.edu/~callan/Projects/IIS-1815528/) describes recent activities and provides access to research publications, experimental results, datasets, and open-sources software produced by the project. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.