Preliminary Algorithmic Foundations for Ranking Quizzes and Students from Student-sourced Quizzes

Carnegie Mellon University, Pittsburgh PA

Investigators

Abstract

The project addresses a new set of problems that will increasingly arise with a gradual shift to massively open online courses and large classes where new assessments to test active learning of the concepts are costly to design. The context of the project is a prototype cloud-based quiz management system that leads the online learners through scaffolded activities, such as creating, answering, and improving of student-sourced quiz items. The system then automatically evaluates the competence of learners and the quality of questions from outcomes of these activities. This project addresses the key algorithmic issue in such systems of efficiently determining the quality of the students and the submitted items. Preliminary ideas for new algorithms will be developed and deployed in the test system to understand their advantages and shortcomings. The development and testing of algorithms will provide opportunities for training two graduate students. This will also lead to a refinement of the eventual algorithmic design of a system for scalable online assessments with significant societal impacts. In particular, the project develops new algorithmic frameworks for problems in the joint ranking of students and student-sourced quiz questions based on feedback in the form of peer performance and review. A key contribution is the formulation of these problems in terms of abstract problems amenable to both combinatorial and continuous optimization, which in turn allows a completely unrelated subfield of people (those of optimization) to work on these problems in the future. At the same time, the project will undertake the groundwork of developing the first such algorithms that solve these problems with current approaches.