EAGER: Collaborative Research: Malleable Media to Support Interaction through Bi-Directional Touch Displays

Regents Of The University Of Michigan - Ann Arbor, Ann Arbor MI

Investigators

Abstract

Interaction with touchscreens is compelling because, in part, it replicates interaction with objects in our environment; you can use a touchscreen to visually apprehend, reach toward, touch and manipulate an object. Of course, this is only partly true because, at the last moment, your hand contacts glass and the rest of the manipulation is simulated. For many applications, the glass barrier is an acceptable compromise because the simulated response of the on-screen application to your action (delivered visually) is sufficient to close a perceptual gap, to fool you into believing that you are actually touching a button or icon; the visual feedback serves as proxy for the missing tactile interaction with the object. However, in some situations, and for some users, the presence of the glass barrier causes the interaction to fail completely. For blind users in particular, for whom closing a perceptual loop through vision is not an option, other means of interaction with computationally mediated environments must be developed. The primary objective of this exploratory project is to develop a full-page interactive tactile display that can render what the PIs term "malleable media" by combining touch sensing with their existing microfluidic actuators to create a full-page responsive surface. Such a display could provide an immediate tangible response in reaction to how it is being touched. By transcending the glass barrier of the screen, such a display would quite literally support a direct manipulation paradigm by making both interface elements, such as buttons and icons, and application content touchable. Thus, project outcomes are likely to provide greater accessibility to digital media by blind users. But the sighted community also stands to benefit. Both students and professionals working in STEM fields require access to increasingly high dimensional data that cannot be easily accessed via speech. An interactive tactile display such as that which is the focus of this work would provide such access, to blind computer users and their sighted counterparts alike. The intellectual merit of this exploratory project derives from combining two strands of existing research by the PIs. The first is their recent work on a microfluidic tactile display, while the second is their established record in applying principles of enactive cognition to the design of human-computer interfaces. Enactive cognition posits a tight coupling between our actions on the environment and our perception of how the environment is responding. With the new technology the PIs have the means of creating such an interactive surface, but it is not yet known what such interactions should "feel" like. What would it be like to sculpt an object by molding the surface of a tablet computer with one's hands? For blind users, this would provide a means for not just feeling tactile objects, but also for creating those objects. Such objects might be used as new ways of representing, for example, patterns present in complex data sets or of interacting with mathematical models.