EAGER/Collaborative Research: Demonstrating the Importance of Research Setting Representativeness in Systems Engineering and Design Research

Purdue University, West Lafayette IN

Investigators

Abstract

This EArly-concept Grant for Exploratory Research (EAGER) grant supports fundamental research into the factors that determine whether a research setting is sufficiently representative of real-world systems engineering and design situations. Designing complex engineering systems such as critical infrastructure, capital-intensive projects, and national defense systems involves large design teams consisting of hundreds to thousands of individuals over timescales reaching into decades. This project addresses the fundamental issue of understanding the design of such complex engineering systems using simpler, shorter-duration, and more controlled research settings. The research objective of this project is to explore the creation and validation of standards for research setting representativeness to assess whether and the extent to which findings from one research setting can be extend to others. If successful, this work will transform research in the community by enabling more efficient knowledge aggregation through a rigorous methodological foundation for assessing how well, and in what dimensions, research settings represent their real counterparts. This foundation will enrich empirical and theory-driven research within the systems engineering and design community by improving validity assessment of research outcomes across diverse methodological perspectives. As a crucial component of this project, results will be disseminated to diverse audience through presentations and workshops at community-specific conferences. Dissemination and community-building activities will foster an ongoing conversation around the need for shared standards and consensus for the kinds of knowledge to be gained from diverse research settings and methodological approaches. The core intellectual contributions of this high-risk high-reward work are to identify dimensions of research setting representativeness, understand how they impact design and execution of research studies, and demonstrate a methodology to generate insights regarding representativeness. Two pilot studies will probe varying dimensions of representativeness for a spacecraft design process. The first pilot study will compare the real spacecraft design process to a concurrent design environment with a smaller team working on a shorter timescale. The second pilot study will compare a fully abstracted function optimization game to a more representative game that models the context and technical dependencies of a spacecraft in the optimization problem. The pilot studies will demonstrate the viability of studying the effects of research setting representativeness on designer behaviors. The potential exists that research setting representativeness is impractical to characterize well, which is a principal risk factor for this project. However, the potential reward is equally high. If successful, results from this project are expected to provide the foundation of multi-method, multi-scale research, allowing researchers to interpret, evaluate, and build on each other?s results across different research methodologies. This would transform the research community and enable new discoveries that far exceed current research capabilities. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.