CAREER: Smarter Educational Software through Sketch Recognition

Harvey Mudd College, Claremont CA

Investigators

Abstract

n many technical disciplines, from electrical to mechanical engineering, the simplest and most straightforward way to communicate an idea is to draw a picture. Diagrams are particularly powerful in education when they are combined with computer simulation programs because, unlike paper, simulation programs allow students to explore the behavior of a physical system, such as a circuit or a mechanical device. Unfortunately, the mouse and keyboard interface to these programs prevent students from drawing their diagrams freely, forcing them continually to consult menus to choose pieces of the diagram. The hardware to draw on the computer exists; the bottleneck is the computer's inability to understand diagrams. The PI's central research goal is to construct and deploy computer simulation tools capable of understanding students' hand-drawn diagrams. The results of free-sketch recognition research cannot yetbe incorporated into end-user applications for two reasons. First, free-sketch recognition is not sufficiently robust to incorporate into useful tools. Second, little is known about how to build usable interfaces that incorporate free-sketch recognition. This work will bridge the gap between free-sketch recognition technology and its end-users by focusing on its application to undergraduate engineering design. The outcome of this work will be improved techniques for free-sketch recognition, guidelines for incorporating free-sketch recognition into usable interfaces, and educational sketch-based simulation tools. This project also includes a significant outreach component aimed at establishing a community of educational and technological researchers with the combined expertise necessary to evaluate the effectiveness of these tools on learning. This research project directly supports the PI's educational goal of increasing the number and diversity of students who use and who develop technology. First, technology that can understand hand-drawn diagrams will provide students with a more familiar interface to simulation programs, lowering the barrier to using technology in the classroom, particularly for students with less computer experience including women and underrepresented minorities. Second, this research will be performed at Harvey Mudd College, an exclusively undergraduate institution. The small size of the college and low faculty to student ratio (1:9) will enable the PI to work closely with undergraduates, teaching them the basics of performing research and preparing many for graduate work.