CRII: SHF: Graphical User Interface Test Code Adaptation for Developing Voice Assistant Features in Mobile Applications

Villanova University, Villanova PA

Investigators

Abstract

The popularity of Voice Assistant (VA) has skyrocketed in recent years, with users able to perform tasks faster and more efficiently through simple voice commands. The success of leading products like Google Assistant and Amazon Alexa has motivated mobile app developers to integrate voice assistant features into their applications. However, adding these features can be significantly time-consuming for developers, as the existing code design primarily focuses on graphical interactions rather than voice interactions. This project aims to provide a solution to this issue by utilizing existing test code to design the voice assistant features. Test code is a series of simulated user interactions with an application that is used to validate the functionality of the application. By reusing the test code and mapping it to the corresponding voice query, the developers can quickly incorporate voice assistant capabilities into their applications. As a result of this project, developers will be more likely to redesign their existing code to accommodate such new voice interactions. Furthermore, this project will bring many training and educational opportunities for both undergraduate and graduate students. It has the potential to support thousands of mobile developers in adding voice assistant options to their applications. In this project, the investigator aims to answer the question of whether and how existing graphical user interface (GUI) test code can be adapted to develop voice assistant features in mobile applications. The investigator will begin by extracting the general functionality from a specific functionality instance in a GUI test code. The investigator will study how to differentiate the general code and non-general code in GUI test code, and develop automated techniques to detect them. The investigator will also investigate how to convert the general functionality into a reusable code template for voice assistant feature development. Second, the investigator will study how to represent the code template using human-understandable language and create potential voice query templates based on this language. A map will be created to connect the voice query template with the code template. Finally, the investigator will develop techniques to facilitate mobile applications in executing the code template in response to a particular voice query. This project aims to support developers in adding voice assistant functionality to existing mobile apps by reusing the test artifacts of the application. The results of this research will inspire further investigation into other forms of code reuse and migration for voice assistants or other new features in mobile applications. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.