RAPD: Analytical and Exploratory Approaches to Communicate Mathematics to Visually Impaired Students

New Mexico State University, Las Cruces NM

Investigators

Abstract

PI: Pontelli, Enrico Proposal Number: 0754525 The human visual system is characterized by a large spatial bandwidth and concurrent processing capabilities, enabling individuals to learn from individual components, groups of items, and patterns in complex visual layouts. The enabling capabilities of visual information processing are vital in many areas the understanding of documents with mathematical content is one of such areas. The traditional representation of mathematics is inherently spatial and multi-dimensional (e.g., matrices, fractions), and it often combines analytical presentation (e.g., formulae, definitions) with intuitive graphical descriptions (e.g., diagrams). This creates a fundamental obstacle for people with visual disabilities; their lack of visual access prevents them from accessing traditional representations of mathematical content, even when encoded in digital format (e.g., on-line mathematics textbooks). The traditional assistive technologies employed by students with visual disabilities (e.g., Braille displays, screen readers) are inherently linear, removing contextual references and imposing very high load on the reader's working memory. Effectively, accessibility of mathematics has become a digital barriers to information and learning opportunities. This is particularly severe in the educational context, where the emphasis on distance education and on-line material has not been properly matched by the adoption of accessibility and universal usability principles. The goal of this project is to develop a body of knowledge and tools that will enable students with visual disabilities to access on-line educational material with significant mathematical content. The aim of this work is to enable visually impaired students to access and learn mathematics with a level of efficiency and following cognitive processes that are at par with those used by sighted students. In particular, we aim at creating a comprehensive framework which provides access to mathematical content at the analytical level (e.g., formulae, theorems, proofs), at the exploratory/intuitive level (e.g., content exploration, access to diagrams and charts), and at the practical level (e.g., problem solving, exercises). The guiding principle of this work is to understand and formalize the underlying semantic structure present in mathematical documents, starting from its basic components (e.g., individual formulae, components of a graph) and moving towards more complex organizations (e.g., logical steps, proofs, theorems, diagrams). The semantic structure is employed to guide multi-modal presentation schemes, which provide an automatically tuned balance between presentation (i.e., continuous access to the content) and navigation (i.e., user-directed traversal of the content), presentation adaptation based on user models, and combination of aural and haptic presentation. Haptic presentation relies on a novel hardware device (the haptic finger) to be designed and developed as part of this project. The framework we propose will also enable visually impaired students to interleave \reading" mathematics with \practicing" it, through an accessible workbook, which directly interacts with the reading process. Different levels of evaluation will be applied, involving visually impaired students early on in the design, and relying on formal cognitive models to assess progress. The project assembles a team of experienced researchers with expertise in the areas of applied cognitive/perceptual research, assistive technology software and hardware research, and usability design and evaluation. The researchers have collaborated previously, e.g., Tran and Pontelli conducted NSF-sponsored research in accessibility of the web for visually-impaired students. Intellectual Merit: (1) Observation and analysis of human-computer interaction by students with visual disabilities with mathematical content presented in electronic form, (2) Development of methodologies to extract semantic representations of mathematical content; (3) Design of a haptic device and control software for the presentation of graphical representations of mathematical concepts; (4) Study of approaches to aural presentation and navigation of mathematical content; (4) Integration of haptic and aural presentation and navigation of mathematics, driven by user models, and composed with workspaces for accessible practice of mathematical concepts; (5) Usability evaluation with sighted and visually-impaired students. Broader Impact: (1) promote teaching, training, and learning, with focus on diversity due to visual disabilities; (2) broaden the representation of students with visual disabilities in their ability to access and navigate documents with heavy mathematical content (e.g., math textbooks), thereby increasing their educational opportunities in all areas, but especially in the sciences and engineering; (3) enhance infrastructure for education by improving the ability of usability experts to evaluate haptic and auditory interfaces in an educational setting. Ultimately, society will benefit from the greater inclusion in education and improved training opportunities for students with visual disabilities and generally by making information more widely accessible to these users. All PIs are actively involved in initiatives to advance the educational opportunities for a diverse student population - e.g., NMSU is a Minority and Hispanic-serving Institution, and students from traditionally underrepresented groups will be involved in the research and educational activities. Keywords: accessibility of mathematics, haptic navigation, visual disabilities, usability evaluation methods