CHS: Small: Compounding Dividends on Voice Banking

Alfred I Du Pont Hospital For Children, Wilmington DE

Investigators

Abstract

Text to speech (TTS) synthesis has become a successful and ubiquitous technology. The area of application for TTS technology that motivates this research is its use for Augmentative and Alternative Communication (AAC). According to the American Speech-Language and Hearing Association (ASHA), more than two million people in the United States have severe communication disorders that impair their ability to talk. AAC devices that use TTS to create spoken output are used by many of these people to support communication. Historically, AAC users have had access to a relatively small family of generic TTS voices that are neither unique to them nor typically age- or dialect-appropriate. However, advances in TTS technology make it possible to create personalized synthetic voices that capture the unique vocal identity of AAC device users if they are able to record enough speech. This allows patients with neurodegenerative diseases such as ALS to "bank" their voice - that is, to record examples of their speech that can later be used to create a personal TTS voice - before the disease progresses to a point that they can no longer speak. Unfortunately, one major barrier to voice banking, especially for patients who may already be experiencing some difficulty speaking, is the amount of speech needed to create a natural sounding TTS voice that fully captures the vocal identity of the voice banker. To reduce this barrier, this research will combine a type of speech synthesis called parallel formant synthesis that was developed several decades ago, with deep learning computational techniques that allow a computer to learn how to control the parameters of the parallel formant synthesizer to reproduce the speech of a target speaker given examples of the target speaker's speech. A parallel formant synthesizer will be implemented and trained to model speech recorded by voice bankers, and its output will be compared with that of other synthesizers that have been trained with the same speech data. Objective measures of similarity between synthetic and natural utterances, and subjective measures of voice quality and similarity using human listeners, will be used. This will be the first step toward building a parallel formant synthesis-based voice conversion system capable of creating TTS voices from a small number of natural speech samples, and also better able to model the expressive nature of natural speech. Despite advances in TTS technology, there are multiple challenges to the application of this technology for voice banking. Specifically: (a) the amount of speech required (several hours) to create the most natural sounding TTS voices using unit selection or hybrid DNN/unit selection is prohibitive for most voice bankers; (b) existing voice conversion techniques that do not require large amounts of parallel speech from the target talker generally produce speech sounding less natural and less like the target speaker when compared to concatenative synthesis; and (c) both concatenative and statistical parametric techniques produce speech that is only as expressive as the data within the speech corpus from which they have been constructed or trained. Parallel formant synthesis, because it is based explicitly on the perceptually most salient features of natural speech and lends itself to independently modeling laryngeal, suprasegmental, and segmental features should be better able to address all three of these challenges. As proof of concept, a parallel formant synthesis (PFS) vocoder with DNN-based parameter estimation will be implemented. The vocoder will be implemented within the Merlin DNN synthesis framework so that speech output of the PFS system can be directly compared to output generated by the World and MagPhase vocoders. Training will be based on corpora drawn from the same set of 1600 utterances recorded by multiple individuals who have contributed their recordings to the ModelTalker project. The selected target talkers will be balanced for gender and span a wide range of English dialects, but use of speakers with noticeable levels of dysarthria will be avoided. Objective comparisons will be based on Mel-Cepstral Difference (MCD) between synthetic and natural sentence tokens that were not used in training the synthesizers. Subjective measures (Mean Opinion Scores) will be obtained from human listeners via Amazon Mechanical Turk. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.