EAGER: Scalable Crowdsourced Reinforcement of Robot Behavior

University Of Vermont & State Agricultural College, Burlington VT

Investigators

Abstract

This project will develop the groundwork for systems that allow large numbers of volunteer contributors to help train robots with different movement capabilities to respond correctly to commands, answering research questions both about how to design the systems to motivate people to contribute and about how crowds can help train robots. This is an important robotics question because training these robots requires large, labeled datasets in which the robots are moving and their motions are appropriately labeled; collecting volunteer contributions ("crowdsourcing") is a potentially cost- and time-effective way to generate these kinds of datasets. To test this, the research team will extend an existing platform that allows people to talk and collaborate around video streams in order to attract and retain participants interested in helping train robots. They will present volunteers watching their stream with simulated robots that are increasingly complex over time, asking volunteers both to suggest appropriate commands that might work well with a particular robot's structure and to give the robots feedback about how well they carry out the commands. Although at first the robots will simply try different possible motions, volunteers' feedback will help the robots learn which motions help them carry out commands, while providing the volunteers direct experience with robotics research that might increase their interest in and acceptance of robots in the future. To encourage long-term participation, the team will also add a number of interface features to the stream that allow volunteers more control over the robots being displayed as well as information about the effectiveness of their training efforts. The research will result in insights into robot learning, the production of large datasets that could be used in other robotics research, and the validation of strategies (such as gradually increasing the complexity of tasks and the demonstration of progress) for encouraging people to participate in crowdsourcing systems. This, too, is an important general question, as crowdsourcing is useful in many socially relevant domains, including building open source software, constructing information resources, and supporting citizen science. Finally, the team will make the supporting software and data available to researchers in domains such as linguistics and developmental psychology that are also interested in language acquisition. To do this, the team will leverage an existing prototype system that is based on a live video streaming platform. This system presents a robot, then cycles through a command selection period where the content people type into the chat window is interpreted as potential commands, and a command response period where the robot performs some action and chat input is interpreted as positive or negative reinforcement for robot learning. An initial trial generated much interest and useful data for learning, but contributions rapidly dropped because the robots did not learn in real-time and did not change. To improve learning, the team will integrate robot control algorithms that leverage crowd input to learn appropriate per-command behaviors online, as well as visual indicators of the amount of learning that might incentivize participation through showing the value of volunteers' contributions. To improve variety, the team will slowly evolve the robot's morphology, adding additional appendages and sensors that support the exploration and learning of new commands while helping maintain volunteers' interest. In addition to evaluating the effectiveness of the data itself in supporting robot learning of commands, the team will measure the levels, frequency, and appropriateness of contributions to evaluate whether changes in responsiveness and robot complexity in fact increase people's willingness to participate, as well as hypotheses about how the apparent capabilities of robots affect the ways people choose to interact with them. The long-term goal is to build a large, sustained community around robot teaching that can help evolutionary biologists, social scientists, cognitive scientists, neuroscientists, linguists, and roboticists all pose and answer novel hypotheses around the co-evolution of humans and robots.