SI2: Conceptualization: Dynamic Languages for Scalable Data Analytics

Purdue University, West Lafayette IN

Investigators

Abstract

This planning grant gathers scientific community requirements for a set of capabilities termed Scalable Data Analytics. The project investigates community needs to support scientific discovery by providing an effective interface between extant hardware resources, data sources and repositories, and system software infrastructure. The proposed effort focuses on software environments and tools for data acquisition, management, visualization, sharing, and analysis for the working scientist, which can scale up to massively parallel and cloud fabrics, but, crucially, which can as easily scale down to a single laptop. Software systems for data analytics are integral to the fabric of scientific innovation. The ability to acquire, process, and analyze large amounts of complex structured and unstructured data is at the core of diverse disciplines. While scientists can exploit large repositories of software tools optimized and refined over the years, significant new challenges are posed by the rapidly evolving characteristics of scientific datasets. These challenges are addressed by software systems that enable development of new software incrementally, modification of existing methods, or techniques for integrating pipelines of off-the-shelf components. For such application needs, scientists increasingly rely on dynamic computer programming languages. These languages facilitate interactive prototyping, support rapid development, and can be embedded or used to manage complex scientific software pipelines.