EAGER: Reinforcement Learning of Multi-Party Negotiation Dialogue Policies

University Of Southern California, Los Angeles CA

Investigators

Abstract

Natural-language-based dialogue systems have a dialogue policy that determines what the system should do based on the dialogue context (also called dialogue state). Manually designing dialogue policies can be a very hard task and there is no guarantee that the resulting policies will be optimal. This issue has motivated research on statistical methods to learn dialogue policies. However, this work has mainly addressed two-party dialogue between one computer agent (system) and one human user, and assumed that the behavior of the user does not change over time. This assumption generally holds for simple information-providing tasks (e.g., reserving a flight), where not much variation in user behavior is expected. But this assumption is not realistic for other genres of dialogue, such as negotiation, where users may change their behavior if they realize that their current negotiation strategy does not help them achieve their goals. This EArly Grant for Exploratory Research investigates automated dialogue policy creation for multi-party dialogue (with more participants than just one agent and one user) without making the assumption that the behavior of the user does not change over time. Previous work has used single-agent Reinforcement Learning (RL) for two-party dialogue policy learning. Single-agent RL requires a stationary environment, i.e., an environment (in our case a user) that does not change over time. Thus single-agent RL is not suitable for a non-stationary environment. For this reason, this project explores the use of multi-agent RL for two-party and multi-party negotiation dialogue. Multi-agent RL makes no assumption that the behavior of the user does not change over time, and is designed for situations where the strategy of one agent may affect the strategy of other agents (non-stationary environment). System-user interaction is done using natural language, and the learned policies are evaluated both in simulation and with human users. The advances made in the project, i.e., multi-agent RL algorithms for dialogue policy learning, are encoded in a statistical dialogue management toolkit (to be publicly distributed). Publicly available multi-agent RL algorithms for dialogue policy learning allow broader access to this technology for dialogue researchers and students.