EAGER: Improving Reproducibility of Computing Research using Proxy Workloads

University Of Texas At Austin, Austin TX

Investigators

Abstract

Computing research is very minimally reproduced, debunked, or deconstructed. This project explores the use of miniaturized proxy workloads in improving the reproducibility of computing research. Past research has shown the usefulness of miniaturized proxy workloads in pre-silicon design explorations. By creating improved proxies and by disseminating the proxies and the proxy generator, this project aims to explore usage of proxies from the reproducibility angle. The proxies can enable reproducing studies with reduced effort, conducting studies with a broad range of benchmarks, improving ruggedness/robustness of the study by covering a large workload space, and by offering more opportunities for simulator calibration and validation. A repository with fine-grain proxies, proxies that cover a broad workload space, and directed proxies that validate specific architectural aspects, is likely to lead to adoption of proxies which will make a big stride towards reproducibility. The proposed explorations will likely transform how computing research is conducted and validated. Currently there is minimal effort on reproducibility of computing research, although ACM has recently instituted a reproducibility initiative. The proposed project can make a broad impact in improving the reproducibility of computing research by the use of proxy workloads. Researchers in the community will be enabled to better validate their own work, and reproduce others' work without an overwhelming amount of time/effort. Other barriers towards reproducibility such as lack of accessibility to proprietary benchmarks, lack of benchmarks in many regions of the workload space, etc. can be addressed using miniaturized proxy workloads. The repository created in the project will enable computing researchers to enhance repeatability, robustness, reliability and generalizability of their research findings. Along with the research activities, an educational and dissemination program will be designed to communicate the results of this work to students, researchers, as well as a more general audience, and to include students from underrepresented communities. The research results will be published at the PI's website.