CAREER: Ordered Alignment Methods for Complex, High-Dimensional Data

Harvey Mudd College, Claremont CA

Investigators

Abstract

Digital devices are collecting temporal data at an unprecedented rate, leading to an explosion of time series datasets ranging from biomedical data to sensor networks to multimedia data. Building automated tools to search, classify, detect, and discover patterns in time series data requires ordered alignment methods to determine the similarity between two sequences. However, there is a gap between the usefulness of these methods and the complex, high-dimensional data that make up much of the modern analytic landscape. This project aims to address three key factors contributing to this gap, using music and multimedia data as a challenging testbed. Specifically, the goal of this project is to design ordered alignment methods that are scalable enough to be used in interactive multimedia applications, flexible enough to handle complex, structured multimedia data, and integrated into modern machine learning models. The project will be carried out at a liberal arts college and involve mentoring 10-15 undergraduate research students, developing course-based research experiences, and establishing a research partnership between HMC and a leading audio and multimedia research group in Germany. The project will address three fundamental questions about ordered alignment methods like dynamic time warping (DTW). The first question is, “How can we make ordered alignment scalable enough to be used in interactive multimedia applications?”. This will be explored by developing parallelizable approximations of DTW that fully utilize modern hardware, as well as hashing-based approximations of DTW for very long sequences. The second question is, “How can we make ordered alignment flexible enough to handle complex, structured multimedia data?”. This will be addressed by utilizing compositions of multiple ordered alignment stages and state-based time warping, in which time warping characteristics depend on a latent state. The third question is, “How can we integrate ordered alignment into neural network training?”. This will be pursued by addressing two current obstacles: avoiding the cold start alignment problem through language model pretraining of discretized features, and handling long sequences by utilizing both mini-batch and epoch-level data processing. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.