RI:Small: Capturing, Perceiving, and Rendering of Artistic Skills for Real-time Interactive Creation of Art

Georgia Tech Research Corporation, Atlanta GA

Investigators

Abstract

Today’s rapid technological advances have raised fears that humanity will lose relevance, that automated machines will eventually outpace human physical capabilities, human thought, and even human creativity, that humanity will be on the losing side of the emerging technology gap. This project will provide new methods for human-robot collaboration in artistic expression, as a step toward bridging this technology gap in domains where creativity is paramount. For instance, consider a master painter with decades of practice on canvas; preserving their mastery of skills and creativity is extremely challenging, let alone replicating their masterpieces. The team will address a myriad of technical challenges associated with capturing artist skills, recognizing their creative intents, and robotically generating art in the artist’s own styles and techniques. Calligraphy presents a complex set of technical challenges, as it involves contact of a soft, deformable brush with a paper surface to apply liquid ink. The team will investigate algorithmic solutions for skillful use of such complex artist mediums by robots. The developed robotic systems will be publicly demonstrated in collaborative performances with professional artists; these dissemination activities will demonstrate how a proficient technology could preserve and amplify the creative status of the human artist. The long-term goal of the project is to achieve genuine human-robot collaboration during the creative, artistic process. Progress toward this goal will proceed along four separate thrusts, each focused on pushing the state of the art in a particular direction. The first step is to build systems that, in some sense, understand the artist’s creative process. For the purposes of this project, understanding can be demonstrated by capturing, perceiving, and rendering the dexterous skills of human artists. The artist’s motions will be recorded using motion capture technology (similar to methods used in modern cinema), and the captured data will then be used to design mathematical models that will form the basis for reasoning about the artist’s actions, and for building perception algorithms that can understand the artist’s motion in real time. Hence, the first research thrust, capture, will develop representations of artistic skills, and algorithms to enable the recording of the creation of an artifact via skillful manipulation of a tool. The second thrust, perception, will develop algorithms that can, in real-time, sense the artistic intent of an artist, using learned representations, requiring much less comprehensive sensing modalities. The third thrust, rendering, will develop identification and control algorithms that allow a robot to render artistic actions in different contexts, e.g., after creation of an artwork is captured or in the context of an interactive system. The final step is designing robotic systems that are capable of rendering these skills, with the goal of enabling genuine human-robot collaboration during the creative process. This symbiotic relationship preserves the creative status of the human artist, while providing augmented capabilities to enhance the creation process. Finally, the fourth thrust is to integrate all of these results into a truly interactive system, where both perception and control are leveraged to enable human-robot collaboration in creating novel works of art. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.