EAGER: Prototyping Touch with FLS-Matter

University Of Southern California, Los Angeles CA

Investigators

Abstract

This project investigates techniques and algorithms that enable immersive and interactive 3D displays using swarms of flying light specks (FLS). An FLS is a micrometer drone with one or more light sources to generate different colors and textures with adjustable brightness. Synchronized swarms of FLSs will render objects in a pre-specified 3D volume, an FLS display. An immersive FLS display enables a user to touch and manipulate objects. For example, a user may pick up an FLS illuminated teapot or throw an FLS illuminated rock. The building block of these objects is FLS-matter, 2D (e.g., a square) and 3D (e.g., a pyramid) geometric shapes using FLSs with extendable behavior. FLS-matter may fundamentally transform how researchers think about human-machine interaction. A successful realization of its proof-of-concept prototype will provide essential insights into implementing an immersive and interactive 3D display. The size of such a display may be a tabletop cuboid, a telephone booth, or even a room. With the potential to transform the future of human communication and perception, and the public's interactions with information and data, these displays can potentially change how people work, learn, receive care, and socialize. The project includes plans for K-12 STEM outreach and for mentoring and training undergraduate, graduate, and high school students through this interdisciplinary research. A new graduate level seminar course on holodecks will be introduced. Apart from organizing the First international Conference on Holodeck at Los Angeles, the research team will participate in outreach activities such as Robotics Open House Events in the University of Southern California. FLS-matter is a novel transformative idea that requires a proof-of-concept prototype. This project focuses on creating a prototype that includes design and implementation of physics inspired frameworks that enable a swarm of FLS-matter to react to the force exerted by human touch without causing injury to the user or damaging the FLSs. Since micrometer-sized drones are not currently available, simulation studies will be used in conjunction with physical studies evaluating behavior of and user interaction with macro-sized off-the-shelf drones. The project will investigate the feasibility of a positioning system to identify the location of the FLSs, which in turn identifies the location of user touch. These studies will include design and implementation of both centralized and decentralized algorithms, and an evaluation of their accuracy and scalability characteristics as a function of the number of FLSs. This research will also use a combination of quantitative models in simulation and human subject studies with commercially available drones to quantify the amount of force exerted by a swarm based on its speed, flight pattern, mass of its FLSs, and the number of FLSs used at the perturbation point by a human touch. These user studies with multiple FLSs working together will determine the user’s contact method and a set of reasonable stiffness values for rendering an object. The tested FLSs will also be calibrated to determine thresholds specific to an individual user. Through the design and implementation of the FLS-matter concepts, this project addresses the challenges in mid-air haptics for user interactions with a swarm of flying robots, including failures, limited flight time between battery charges, collision-free flight paths, and coordinated force output. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.