CyberSEES: Type 1: Collaborative Research: High-Performance Image Classification and Search Supporting Large-Scale Seafloor Biodiversity and Habitat Surveys

University Of California-San Diego, La Jolla CA

Investigators

Abstract

Seafloor ecosystems are complex environments populated by a great diversity of organisms. Unfortunately, these ecosystems are increasingly threatened by direct and indirect human activities, including changes in land-use practices, coastal runoff, energy and mineral extraction, and fishing pressure. Developing effective sustainability policies to deal with these ecosystem threats requires that we first understand seafloor communities as they are today, and then track how they change over time as human activities shift and sustainability policies are modified. Recent advances in high-resolution underwater imaging offer new ways to do this. Survey ships can zigzag back and forth above a threatened region, towing a submerged camera system that repeatedly snaps pictures of the seafloor. This produces an enormous and valuable image set that captures the current state of a seafloor ecosystem. Surveys like this have been done for many threatened regions, and more are in progress. Substantial challenges remain to process these image sets. A useful characterization of a seafloor habitat requires knowing which specific types of corals, sponges, starfish, and so forth are present, how many there are, and how they are distributed throughout a region. But with each survey image set containing hundreds of thousands or millions of images, manual processing is impractical. Instead of an army of experts examining these images, computer software can scan each image and automatically recognize the color and texture of different seafloor species. Experimental classification software like this exists today in research laboratories, but the software is slow. To be useful for huge image sets, this software must be revised and optimized to run on the latest high-performance supercomputers. This is the focus of the project, which will yield new optimized classification software that can quickly sweep through enormous image sets to classify and count the species present and provide essential information about the health and biodiversity of threatened seafloor ecosystems, or any other ecosystem with a suitable image set. Then, when surveys are repeated for the same region every few years, this processing can reveal important trends that document the health of a region and the impact of new sustainability policies that aim to mitigate continuing threats to these communities. This project leverages prior work prototyping seafloor image classification algorithms. These algorithms divide survey images into small tiles, then characterize each tile with a high-dimensionality feature vector that includes metrics on the colors and textures present in the tile, along with water temperature, salinity, and depth data collected by the survey apparatus at the moment the image was captured. Colors in the feature vector are chosen based upon a quantized hue histogram of the tile, while textures are characterized by luminance Discrete-Cosine-Transform (DCT) coefficients. A tile's feature vector is then compared against stored feature vectors for known species within a large classification library. A probability-based selection using a set of nearest-neighbor matches from the library yields a best guess for the species depicted in the image tile. This process is repeated tile after tile, image after image throughout an image survey. Classification performance is strongly a function of the classification library size and the dimensionality of feature vectors used for image tiles and library entries. This project's approach to improve classification performance uses a customized k-d-tree search data structure for the classification library, along with domain knowledge to guide and tune the classification process. The project begins with new methods to cull the tree, prior to classification, by using broad survey characteristics, such as the geographic region covered, water temperature and salinity, the sea bottom type from acoustic data, and so forth. Additional techniques optimize the construction and matching of feature vectors by using survey and library metrics to cull and weigh vector components (such as contextual color gamut and texture detail reduction, principal component analysis to combine and weigh features), reduce the nearest-neighbor set size by using k-d tree metrics on library diversity, restructure the k-d tree to improve common case search and cache performance, and parallelize the search for efficient classification across multiple threads, cores, processors, and nodes in a large compute cluster. Together these new methods are expected to substantially increase classification performance and enable efficient processing for the latest large survey image sets.