CAREER: Investigating linguistic and cognitive abstractions for solving word problems in minds and machines

University Of Texas At Austin, Austin TX

Investigators

Abstract

Math word problem solving is a major stepping stone in childhood education. An intellectual challenge in studying word problem solving is separating out and modeling the different cognitive skills that word problems draw on, such as mathematical reasoning, verbal reasoning, linguistic knowledge, world knowledge, and commonsense reasoning. All of these can contribute to the difficulty that children face when solving word problems but it has traditionally been difficult to identify the role that language plays since human language is messy and complicated. With the advent of Large Language Models, of which the most well-known recent example is ChatGPT, it is now far easier to model and systematically probe a wide variety of language behavior. This CAREER project will use Large Language Models and computational techniques to study math word problem solving in elementary school students (Grades 3-5). The project will lead to the creation of a novel data set of word problems, new insights into both word problem solving and language models, and a system that allows for the generation of custom word problems for use in classrooms. The project has the potential to lead to the eventual development of techniques that will help children who struggle with word problem solving. The investigator’s teaching mission focuses largely on teaching students and the public about the increasingly important and rapidly changing landscape of language technology. He will teach courses focused on language technology, ethical issues surrounding language models, and participate in public panels for disseminating this information more widely. To enable studying the capacities required for solving word problems in higher granularity than was possible before the existence of Large Language Models, this project will create a novel bank of word problems that vary along a variety of linguistic dimensions (e.g., syntactic and lexical complexity). Then, the project team will use modern causal intervention techniques on computational language models to partition out the sources of difficulty in these problems. The second phase of the project is to collect human data from kids in Grades 3-5, modeling individual variation and demographic variation along mathematical, verbal, and linguistic dimensions. The third phase is to develop language-model-based system for generating word problems that vary in difficulty along several possible dimensions. The computational modeling work has the potential to answer longstanding questions about how linguistic variables can increase or ameliorate word problem solving difficulty. On the computational side, there is the potential for major improvements in language model interpretability, which is itself a major goal in computational language processing since these models are, in effect, black boxes and the ways they implement abstract behaviors remain mysterious. This CAREER project is supported by the EDU Core Research (ECR) program. ECR emphasizes fundamental STEM education research that advances fundamental knowledge in the field on STEM learning, broadening participation in STEM, and STEM workforce development. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.