RAPID: Harvesting Speech Datasets for Linguistic Research on the Web (Digging into Data Challenge)

Cornell University, Ithaca NY

Investigators

Abstract

Distinctions of prosody (rhythm, stress, and intonation) are ubiquitous in spoken language. It often seems obvious to a native speakers of English what prosody is most appropriate in a given sentence and context, and researchers in Linguistics and related fields have proposed numerous formalized hypotheses about it. But establishing the validity of these hypotheses is remarkably elusive. Much of the problem is that it is difficult to observe enough examples of a given phenomenon to evaluate hypotheses. The project aims to address this problem of a dearth of data by collecting or "harvesting" examples of specific word sequences or word patterns from web sources. It is often possible to find hundreds or thousands of examples of people using the very same word pattern. If these examples are collected together into a dataset and made available to the research community, it will be possible to evaluate theories about the form and meaning of prosody on an unprecedented scale. Scaling up available data can be expected to have a transformative effect on our understanding of prosody. Audio and audio-video recordings of spoken language, including podcasts, radio and television broadcasts, lectures, and much else, are pervasive on the web. This does not help in itself, because it is not possible to listen to tens of thousands of hours of speech in order to find a few hundred examples of a certain type. Fortunately, more sites are becoming available that provide text transcriptions obtained with automatic speech recognition (for instance Fox Business News, WNYC, Elections Video Search at Google, and university lectures at MIT). Industry blogs and newsletters indicate that more large sites will come online soon. By searching for a word pattern in the text transcription and subsequently retrieving an audio or video file, it becomes possible to find relevant data. To construct datasets for prosody research from these web sources, the project team will implement software harvest engines that interact with the web through standard protocols. Datasets for eight to twelve specific phenomena will be collected. In order to demonstrate the impact of a data-intensive methodology, the samples will be analyzed using techniques of statistics and formal linguistics. For instance, an approach known as machine learning classification will be used to identify the specific features of the sound signal (such as pitch, vowel duration, and intensity) that are responsible for the perception of prosody. Prosody and intonation play an important role in making the discourse coherent, in signaling what part of the communicated information is foregrounded and backgrounded, and disambiguating speaker intention. Any advancement in understanding prosody will not only deepen our understanding of the human language capability, it also has implications in a wide range of areas, including language instruction, translation studies, speech therapy, improving comprehensibility of synthesized speech, and improving speech recognition systems.