CAREER: Theory and Algorithms for Learning with Frozen Pretrained Models

University Of Wisconsin-Madison, Madison WI

Investigators

Abstract

Adapting pretrained models to new tasks and new data is crucial for the democratization of modern machine learning. Application of large pretrained models to customized tasks or new data is difficult since full retraining of the model is not possible due to the huge size and inaccesibility of the original training data. The inaccessibility and large size of the training data also renders traditional domain adaptation techniques impractical. To overcome this challenge there has been shift towards modular adaptation methods that enable the fine-tuning of "frozen" pretrained models with minimal or no modifications to their internal parameters. Despite the success of these methods in some practical applications, there is a gap in theoretical understanding of the factors affecting the effectiveness of finetuning of pretrained models. This project develops a systematic framework that will lay theoretical foundations and lead to rigorous fine-tuning principles. The project has the potential to impact a wide range of scientific fields and industries that rely on adaptation of pretrained machine learning models. The project's broader impact includes the establishment of educational initiatives for advancing STEM, contributing to the development of the future workforce in modern machine learning. The project's goal is to establish a unified theory and devise new algorithms with provable guarantees for the emerging paradigm of learning with frozen pretrained models. A mathematical framework will be developed to facilitate the theoretical analysis of the expressive power of frozen pretrained models under various adaptation and fine-tuning methods. The project will investigate three adaptation strategies: (1) parameter-efficient fine-tuning, which updates a minimal portion of the pretrained model's parameters while keeping the rest unchanged; (2) input/output processing, which involves modifying the input entering or the output produced by the pretrained models; and (3) model composition, which constructs a system of multiple pretrained models to address more complex tasks. In addition to analyzing these methods, the project aims to develop novel adaptation algorithms with provable performance guarantees. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.