EAPSI: Developing a Semantic Attributes Learner through Machine Learning Approaches

Kim Diana S, Branchburg NJ

Investigators

Abstract

This project aims to prove that machine learning approaches in computer vision can discover two key components of fine art classification: first, how humans recognize and classify different visual styles for a target object, and second, what semantic visual attributes they use to finalize their classification decision. Working with a large data set of fine art paintings, the project will investigate a computational procedure to identify a list of word descriptions of different visual styles that is interpretable to humans and is further valid to encode all styles of painting. It can be difficult to provide objective grounds that necessarily determine a visual style: even for the art expert, it is not easy to explain why Claude Monet?s Poppies is classified as impressionist based on its attributes. If the computational algorithm automatically finds semantic attributes determining visual styles that are recognizable to human observers, the result will provide scientific analysis of the human visual perceptual process which is known to be complex to specify. After stabilization, the algorithm will generate annotations describing visual styles for a massive image data set without expensive human work. This data set will be useful data set for future computer vision research. This project will be conducted in collaboration with Professor Seung Wan Hwang in the Data Intelligence Lab at Yonsei University in Korea. Professor Hwang has devised qualitative and quantitative methods to find semantic attributes through data pattern analysis. This award supports a research study to design an attributes learner algorithm from datasets that will enable classification of fine art painting styles, and produce extended datasets containing valuable features of the art work that can be annotated automatically via learned attributes generators. Rather than an expensive training set of annotations to learn the attributes of interest, the PI will design a learner which automatically harvests attributes without human supervision. This approach eliminates the need for a predefined (and potentially subjective) vocabulary of semantic attributes which require expert annotation. The project will use numeric high dimensional data gotten through a Deep Artificial Neural Net (ANN) model. The ANN model is trained through a big image data targeting art style inference. With the unsupervised deep architecture that correlates images and textural data through a shared hidden layer, it is expected that the hidden layer?s positive or negative variables will be interpreted as informative attributes. The data set will include some amount of redundant and hard-to-decipher information, so it requires compression and translation to human-interpretable concepts. Since available ground truth information of the data set is limited to authors, year, and art style, cooperative work with the hosting researcher will focus on the extraction of pattern information between the ground truth information and numeric data. There has been similar research work related to feature extraction in academia, but regarding the new subject of Fine Art style and unsupervised attributes learning, this research will be innovative. This award under the East Asia and Pacific Summer Institutes program supports summer research by a U.S. graduate student and is jointly funded by NSF and the National Research Foundation of Korea.