SHF: Small: Open-domain, Data-driven Code Synthesis from Natural Language

Carnegie Mellon University, Pittsburgh PA

Investigators

Abstract

One of the major hurdles in programming is turning ideas into code; all programmers, even experts, frequently reach points in a program where they know what they want to do but cannot easily turn it into a concrete implementation. In this case, it is common to turn to the web, e.g. enter a natural language query, search, browse results, copy-and-paste appropriate code, and modify it to the desired shape. However, this process is still time-consuming. This research aims to automate and enhance this process, by creating new data-driven methods for code synthesis from natural language, which allow developers to go directly from natural language description to code. Specifically, this project's goal is to bring code synthesis to the open domain, moving from highly engineered methods that work on only a single programming language or task, to methods that have the flexibility and scalability to answer most of the questions asked by programmers, in many different programming languages. The intellectual merit of this cross-disciplinary project lies in its potential to contribute to software engineering through the examination of developer's interaction with natural language productivity tools, and its potential to contribute to natural language processing through new models to understand procedural texts. This project will have broader impact through the development of tools and data linking together programs and natural language, potential to improve STEM education by lowering the barriers to programming, and training of graduate and undergraduate research assistants who will be able to straddle and act as bridges between the fields of natural language processing and software engineering. There are three technical pillars to the work. First, it will focus on methods to mine data consisting of natural language and corresponding code at scale, necessary for training. The mining will be performed over existing online data sources, such as community question answering sites (Stack Overflow) and open-source software repositories (GitHub), using machine learning models that consider both content matches and available meta-data, and crowd-sourcing-based verification of the extracted data. Second, the project will develop code synthesis methods that have the flexibility to handle the wide variety of expressions expected across a variety of software projects and developer needs. This will be done by developing models using neural networks, which have recently shown impressive ability to interpret a wide variety of expressions in other natural language processing tasks. We will expand these models to condition on project context, which will ensure handling of the various constraints necessary to create well-formed programs and allow for adaptation to project-specific conventions and needs. Third, the project will develop methods for learning and improving the models from developer behavior, by feeding back corrections to the generated code into the system and learning from the differences between the pre- and post-correction code. These methods will all be integrated into developer support tools that can be used in a development environment, or through an online API. The utility of these methods will be examined in both controlled and in-the-wild studies. Controlled studies will examine the subjective accuracy of the mined data and generated code, as well as the effect of the tools on the efficiency and ease of development, for programmers from novice to expert level. This project will also create and release tools for general consumption, solicit feedback from a wide variety of developers, and examine how developers use the proposed tools. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.