AF: Small: Mechanism Design for the Classroom

Northwestern University, Evanston IL

Investigators

Abstract

Mechanism design studies how the rules of a system can be designed so that good outcomes are obtained when individuals participating in the system are strategic. This project develops -- and initiates the theoretical study of -- a collection of mechanism design problems for the classroom. Specifically, it views the classroom as a computational system where some participants may manipulate the system to obtain better individual outcomes (i.e., the students) and some participants may be unreliable (i.e., the graders). The instructor aims to put in place policies with a number of natural objectives, e.g., optimizing learning outcomes, fairness of grading policies, and efficiency with respect to effort from participants (both students and graders). By understanding the classroom as an application domain for mechanism design, classroom outcomes can be improved. Moreover, a foundation for mechanism design that is grounded in practice can be established. This foundation may have an impact on other application domains for mechanism design, such as online markets. This project explores three main thrusts: fairness in heterogeneous grading, grading to optimize study incentives, and the design of student feedback mechanisms. Thrust 1: Randomizing questions from a large bank of questions is a popular cheating deterrent in online exams. When students are assigned questions with heterogeneous difficulties, however, fair assessment is not straightforward. Some students may be assigned easier questions than others and the simple averaging of scores will favor these more fortunate students. A performance benchmark for fair and accurate assessment is a student’s average grade on the full question bank. This project aims to (a) develop grading algorithms that perform well with respect to this benchmark for any assignment of questions to students, and (b) understand the impact of the structure of the assignment of questions to students on performance, i.e., exam design. Thrust 2: When students are assigned tasks, their level of effort depends on how their effort is graded. Effort can result in learning; however, it is not directly observed. For example, an article reading task might be assessed via reading comprehension questions where answers to these questions can be assessed, but the amount of effort of reading cannot be observed. The grading of knowledge can be understood in the paradigm of scoring rules. Scoring rules are a classical paradigm for incentivizing a forecaster to report a prediction about an unknown state. In the classroom context, student answers to questions about course material can be interpreted as a prediction of the correct answer. In this context, this project aims to develop a theory for the optimization of scoring rules, i.e., identifying scoring rules that incentivize the students to exert effort (that results in learning). Thrust 3: Feedback to students enables them to assess how their effort leads to outcomes that are relevant to them, such as learning or grades. For example, less precise feedback on grades could lead to more consistent effort because it would avoid students overreacting to spurious high or low grades (cf. overfitting in machine learning). Information design is a classical paradigm where a principal signals an agent about an unknown state to entice the agent to take a more favorable action for the principal. The project aims to understand the sequential provision of feedback to students as a problem of information design and to design good feedback mechanisms. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.