CAREER: Active Learning in the Real World

Florida State University, Tallahassee FL

Investigators

Abstract

This award is funded in whole or in part under the American Rescue Plan Act of 2021 (Public Law 117-2). The explosive growth of digital data in the modern era has expanded the possibilities of solving real-world problems using computational learning frameworks. However, annotating the data with labels, which often includes manually adding informative tags on variables of importance in order to train a machine learning model has remained an expensive process in terms of time, labor and human expertise. Active learning algorithms alleviate this challenge by automatically identifying the salient and exemplar samples from large amounts of unlabeled data, which need to be labeled manually. In a traditional active learning setup, the labeling entities, called oracles, are assumed to be infallible; that is, they always provide the correct answers (in terms of class labels) to the queried samples. However, in real-world applications, the oracles are often imperfect; they may provide incorrect annotations and may even be reluctant to provide any annotations. The overarching goal of this project is to develop novel active leaning algorithms under such challenging, real-world constraints, in an effort to bridge the gap between ideal world and real-world active learning. This project has the potential to tremendously reduce human annotation effort in the design of real-world AI systems (such as medical diagnosis, security and surveillance), thereby creating a significant societal impact. As part of this project, the investigator plans to educate a wide spectrum of students which will help to ensure a strong pipeline of computer scientists to satisfy the technical needs of the nation; he will also seek to increase the number of students in graduate STEM related programs. The goals of this project will be realized through two research objectives. First, a novel active learning framework will be developed, which can jointly identify the informative unlabeled samples together with the optimal labeling oracles for each sample. This should maximize the probability of obtaining the correct label. Second, novel query and annotation mechanisms will be explored which are less error-prone and easier to answer, thereby increasing the chances of obtaining reliable annotations. The second objective will be demonstrated through two innovative query frameworks: (i) a deep active learning algorithm for vision-based facial age estimation, where the annotators merely need to provide the best estimated upper and lower bounds on the age of a person within a given span, rather than the exact age which may be difficult to estimate; and, (ii) a deep active learning framework for multiclass classification, where the human annotators are allowed to provide alternative labels in response to a given query apart from the topmost choice. This is useful in applications where the annotators are not cognizant about all the classes and may have more than one class choices in mind for a given sample. Extensive user studies will also be conducted to understand the benefits and drawbacks of the proposed solutions. This research will open the door to the development of novel active learning frameworks which are designed to operate in the presence of challenging, real-world constraints. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.