EAGER: III: Learning with less data: Capitalizing on formal pedagogies and human performance to incorporate domain knowledge into deep learning models

Cuny Brooklyn College, Brooklyn NY

Investigators

Abstract

Humans are able to learn with greater efficiency than machine learning models, in large part because they learn not just from exposure, but also from domain knowledge, which includes codified knowledge and guided practice. This project will develop new approaches for integrating domain knowledge into deep learning models. It will create models that can be trained with less data as well as mitigate data biases (e.g., data collection that is skewed towards inducing a particular pattern that is not necessarily reflective of the range of ways humans perform a given task). This research will be explored within the musical domain, as it has rich pedagogical and performance traditions for skill generation that can be leveraged in model development. In addition, working with music is an excellent testbed for developing models that can be applied to other domains. For example, there are direct parallels between music and language in terms of pedagogy and practice. Broadly, the models developed in this project will have utility for scientists interested in modeling domains that are data-poor, but expertise-rich as well as counteracting known biases in training datasets. This work also has the potential to foster the participation of a wider range of scholars in computer science research, as expressions of their domain expertise would be more relevant to model development. This project will demonstrate the value of incorporating domain knowledge into structured prediction for temporal deep learning models in complex domains using distillations of established pedagogies and expressions of skilled practice. Its goal is to help machines learn more efficiently by mimicking the ways in which humans learn, as well as to develop models with increased accuracy and interpretability. A central hypothesis underlying this project is that the types of pedagogies that are useful for efficiently teaching humans are also useful for teaching machines. The project examines the research hypothesis through the task of reducing complex musical signals, i.e., digital representations of musical sound, into their essential structural components. Musical signals are particularly challenging to perform this type of reduction on because they are complex temporal signals with a metrical structure. Thus, they are a useful testbed for developing machine learning models for broader applications, most directly in natural language processing but also in other domains with complex temporal signals such as earth science and economics. The task of reducing musical signals will be addressed through three main sub-tasks. The first is model development, which will involve systematic experimentation while integrating domain knowledge as constraints in adversarial networks. The second is domain knowledge encoding, which will establish best practices for encoding pedagogical expertise and performance practice into a machine-readable format. And the third is an exploration of how this music-specific work can be applied to natural language understanding specifically and ultimately formulated as a generalized framework for integrating domain knowledge into deep learning models. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.