Advances in Computerized Adaptive Testing: Modeling Response Times and Constraint Management for Skills Diagnosis

University Of Illinois At Urbana-Champaign, Urbana IL

Investigators

Abstract

Computerized Adaptive Testing (CAT) has become popular in high-stakes testing programs. Examples of large-scale CATs include the Graduate Record Examination, the Graduate Management Admission Test, the National Council of State Boards Nursing, and the Armed Services Vocational Aptitude Battery. An advantage of CAT over paper-and-pencil exams is that it provides more efficient estimation of abilities because it can appropriately tailor item selection to the estimated abilities of examinees. This research addresses two areas of critical importance for CAT. First, flexible models for response times will be developed to assist in controlling the duration of an exam, and also to assist in the measurement of ability in appropriate circumstances. Second, statistical methods for constraint management will be developed to ensure that an exam has sufficient information to diagnose fine-grained skills while also providing an accurate summary score. The impact of the research will be to provide technology to better utilize response-time information and also enhance the ability of CAT to provide diagnostic information. Models for response-time distributions will be developed that make few assumptions concerning functional form and allow for dependence between response times and a latent trait that represents ability on the studied domain. Estimation techniques will be developed that may be used with data previously collected from exams administered with CAT. Algorithms for utilizing estimated response-time distributions will be constructed to better manage duration of exams and to extract information from response times to better estimate the ability the exam is designed to measure. In addition to addressing response times, the problem of managing the diagnostic information to assess mastery of fine-grained skills will be studied for exams that also aim to provide a single summary score. CAT methods for adaptively selecting items to more efficiently provide a score will be modified to simultaneously balance the coverage of skills and attributes of interest.