I-Corps: Automating the Development Process of Mobile Applications

University Of Texas At Arlington, Arlington TX

Investigators

Abstract

Developing a professional mobile phone application is cumbersome and expensive in practice. To create for such an application an optimized user interface, professional graphic artists (i.e., non-programmers) design each screen of the user interface, for example, as pixel-based designs in Photoshop. Programmers then manually convert these screen designs to application source code. This conversion is labor-intensive and typically needs to be done several times during a project, as during a project user interface designs are often optimized iteratively. This project explores the commercialization of the first technique to automatically perform this conversion from pixel-based screen design drawing to working user interface source code. The project is expected to benefit both professional mobile application developers and students. The tool may save professional developers time and teach students how screen elements can be implemented in source code. To satisfy consumer demand for customized user interface (UI) designs, mobile application UIs often deviate from standard UI components and provide their own UI designs. While many graphical UI builder tools exist, they are not well suited for such custom UI designs and expert graphic designers prefer pixel-based design tools. On a given screen design graphic, the project team's prototype tool identifies UI elements such as images, text, containers, and lists, via computer vision and optical character recognition (OCR) techniques. The prototype further infers a suitable UI hierarchy and exports the results as source code that is ready for compilation and execution on mobile computing devices. The generated UI closely mimics the UI of a corresponding real application, both in terms of pixel-by-pixel screenshot similarity and by the similarity of their runtime UI code structures. During the I-Corps program, the team will focus on contacting potential customers, gathering their feedback, and using it to further enhance the proposed prototype.