Emerging Research- Empirical- Development of an Observation Tool for the Informal Science Field:Refinement, Field-testing and Establishment of Psychometric Properties

Mclean Hospital, Belmont MA

Investigators

Abstract

This research project plans to refine, field test and establish the psychometric properties of a formal assessment tool, Dimensions of Sucess (DOS) for STEM Learning, in out-of-school time (OST) settings. With the help of this tool, formal observations on various dimensions of programming and learning of informal science can be conducted. The goal of the project is to assess the tool's psychometric properties, primarily its construct validity, so the instrument can be used widely to assess the program quality. Phase 1 will focus on iteratively refining and revising the DOS for STEM learning, building on existing experience with the protocol. Reliability analysis on the instrument will be condusted using a sample of educators. In Phase 2, student and teacher self-report surveys will provide additional data, which will be analyzed to support the further validation of the instrument. Observations of 100 OST educators with their students on two occasions by two randomly assigned observers will allow for full analysis of the tool and for the development of the training program for observers during Phase 2. The project will have significant implications for the quality and evaluation of OST programs. At the local level, DOS will enable practitioners and administrators of programs to improve program quality and professional development. The project will yield a valid instrument that can serve as a standard for assessing OST STEM programs to improve program quality and to compare programs across sites. Thus, larger studies of OST STEM programs will be possible, contributing to the shared knowledge base. The program will offer training via the internet to program staff, researchers and evaluators, and will provide support for the use of the instrument, so that the instrument can be used widely. The training support will build capacity for program development and evaluation in the field of informal science learning.