Development of a Generic Portfolio Review Tool

American Institutes For Research In The Behavioral Sciences, Arlington VA

Investigators

Abstract

The purpose of this grant is to develop a portfolio review tool (PRT). The PRT will be a computerized tool with a set of coding sheets that guide reviewers through a targeted set of decision rules and questions to record information relevant to a program's portfolio. The primary goal of this exploratory project is to develop a generic, versatile, and multipurpose tool that helps evaluators conduct a systematic and comprehensive review of projects designed to advance STEM education. As part of the PRT development effort, the PIs will first develop a unifying logic model for STEM education and a set of common metrics that are designed to guide and inform the overall process of portfolio review. Additionally, a set of common metrics will be developed to assess progress toward desired outcomes (e.g., whether long-term goals have been accomplished, whether investments yielded products and impacts). Next, they will develop a prototype PRT with the Microsoft ACCESS database system. It will be pilot tested for usability on a sample of up to six programs across directorates that vary in several portfolio dimensions (e.g., evaluation, broad impact). After revising the prototype tool on the basis of the results of the pilot test (e.g., user interface, inter-rater reliability), the PIs will convert it to a Web-based tool with basic functionality (e.g., data entry, search) and design features. Lastly, a field test will be conducted on a different sample of up to three programs.