Evaluation Capacity Building Virtually - An ELearning Tool for Program Planning and Measuring Program Outcomes

Health In Color, Inc., Eliabeth City NC

Investigators

Abstract

ABSTRACT EVALUATION CAPACITY BUILDING (ECB) - VIRTUALLY Lawmakers and the public have pushed for increased accountability in health and social programs. As a result, evaluators and evaluation skills have become very important to both large and small organizations providing health and social service programs. However, much of the public health workforce is not well equipped to handle these increased demands for outcomes data and research. Many federal agencies (e.g., Health and Human Services, Department Labor, Department of Justice) and private foundations are beginning to fund only organizations that deploy evidence-based interventions demonstrating impact on behaviors (e.g., individual, organizational, community). Our firm, through its national evaluation work has witnessed, first hand, the challenges faced by the staff and leadership of many organizations engaged in planning and responding to health issues, including (1) lack of expertise in evaluation and the ability to capture data related to program outcomes; (2) adverse experiences with external evaluators and researchers; (3) lack of resources, limited time and inadequate funds for independent evaluations; and, (4) fear of negative evaluation results. These inhibitors are also cited in literature and reports on evaluation capacity building. Evaluation can require considerable time and resources and many organizations cannot afford to create separate evaluation departments or hire external evaluators. EVALUATION CAPACITY BUILDING - VIRTUALLY (ECB-V) is a product concept that has emerged from our firm?s 23 years of work in program planning and evaluation with government and nongovernment organizations. We envision ECB-V as customized, multimedia Elearning course (ECourse) with a supplemental EAssistant to transfer learning from the individual to the program and organizational levels. The goal of this multi-level Elearning and engagement tool is to improve the capacity of the program staff of community-based organizations and state and local health departments to plan, implement and evaluate programs that advance population health in the U.S. The REESSI staff will focus on two key aims in carrying out this project ? (1) Our team will develop the ECB-V prototype through iterative activities with experts and end-users and (2) evaluate the ECB-V prototype using quantitative and qualitative methods. The transdisciplinary team of population and public health experts, evaluators and software developers will engage in this proposed project to develop the prototype and conduct a feasibility study to determine the degree to which program staff and independent evaluators find the ECourse and EAssistant appealing, usable and valuable (feasibility) and how the ECourse and EAssistant affect program staff knowledge of program planning and evaluation (merit). The development of the user process for the product will be guided by the Evaluation Capacity Building Theory of Change Model to design and deliver stimulating and interactive content with electronic surveys in a digital environment. ECB-V will consist of a comprehensive ECourse with eight learning modules and an EAssistant for on-the-job tasks. The proposed topics (content) for the ECourse are based on (1) the Centers for Disease Control and Prevention Evaluation Guide and (2) the Office of Minority Health Evaluation Protocol. The Phase I activities will provide the foundation for the construction of the full and final components of the ECourse and EAssistant. The Phase II activities will consist of a multi-site randomized controlled evaluation that focuses on both individual and organization outcomes (e.g., attitudes, evaluation knowledge, public health core competency skills, reported changes in organizational systems and decision making).