RI:Small: Nonlinear signal representations for speech applications

Johns Hopkins University, Baltimore MD

Investigators

Abstract

Human speech is a very rich signal. In addition to words, it contains several kinds of important information about the speaker such as identity, gender, age, native language, dialect, and emotion. It also provides information about the transmission channel and environment; for example, whether the speech came from a phone call or a high-fidelity recording, and whether or not there was background noise. This project aims to create a powerful uniform representation that reflects all the information carried by speech. Such representation would enable important speech applications in multiple sectors of society: commercial (security, healthcare, user interfaces), government (security, information filtering), and law enforcement (speaker identification, forensics). In this project, Johns Hopkins University researchers, who invented the original i-vector framework, intend to progress beyond the linear i-vector approach by investigating non-linear models with the expectation to better explain the complex structure of speech. To achieve this goal, two different models are investigated. First, a non-linear i-vector version is explored. In this method, the speech signal distribution is modeled by a Gaussian mixture model (GMM). The super-vector formed by the GMM means is a non-linear function (neural network) of a latent variable (speech representation). The parameters of the neural network and the latent representation can be jointly estimated by stochastic gradient descent iterations. Secondly, the team intends to investigate different types of auto-encoder networks (AE, VAE, RBM, DBM) to obtain representations from their hidden layers. Preliminary research shows that it is feasible to obtain good representations by combining activations from several hidden layers. Visualization tools are used to understand how the speech data have been represented and structured. By understanding the non-linear relationships created via the auto-encoder network modeling and using the visualization tools, there is potential to produce valuable insights into speech modeling. These insights can help cognitive science and neuroscience researchers to understand how the brain represents speech signals. The proposed methods are developed as software that takes a speech segment as input and generates a single vector that may be used to characterize the segment for the important applications mentioned in the previous paragraph. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.