IIBR Informatics: Automatic generation of multi-layered, information-rich 3D maps of ecosystems from images

University Of Georgia Research Foundation Inc, Athens GA

Investigators

Abstract

Ecosystem maps provide information on the abundance, distribution, and spatial relationships between species and their environment, providing insight into the fundamental ecological processes (competition, predation, facilitation, etc) that generate observed patterns. Remote imaging from satellites and airplanes have long been used to map the distribution of ecosystems and habitats, but typically the imagery from these sources does not have sufficient resolution to identify individual organisms limiting their use for studying populations and communities. This project will build on recent developments in computer vision and artificial intelligence that allow 'local' images taken with consumer-grade cameras to be assembled into three dimensional maps of ecosystems. Tools, based primarily on deep-neural networks, will be developed to automatically map the distribution of species and substrates, individuals, and biological communities. These newly developed approaches will enable novel insights into spatial patterns and relationships among organisms by providing rapid, automated methods to map ecosystems, in contrast to current manual, lab-intensive mapping methods. The project will support several graduate and undergraduate students and will involve citizen scientists of all ages through web-based games. Spatio-temporal patterns in ecosystems are the product of fundamental ecological processes such as environmental variation, resource limitation, competition, and predation. In this project, tools will be developed for high-resolution (mm to cm) mapping of ecosystems using local imagery allowing rapid assessment of the abundance and distribution of organisms, and spatial relationships amongst organisms. These tools will integrate recent advances in deep learning architectures such as convolutional neural networks (CNNs) and algorithms for assembly of 'local' images to automatically generate three-dimensional (3D) information-rich maps of ecosystems. Commercial and open-source software are currently available for automated assembly of overlapping 2D images into 3D reconstructions of the imaged scene. The methods develop here will overlay relevant biological information encoded in the corresponding images upon 3D reconstructions using CNN based image analysis methods. The specific goals of the project are: 1) semantic segmentation of the reconstructed 3D maps, i.e., labeling of each point on the 3D map as one of a number of predefined classes, 2) identification of individuals via object detectors, 3) mapping of communities using unsupervised topic models, and 4) incorporation of moving organisms using non-rigid structure from motion, resulting in an information-rich, multi-layered 3D map of the ecosystem. The tools developed in this project will be generally applicable to most ecosystems; however, coral reefs and salt marshes will be used as exemplar ecosystems with relevant contrasts in biodiversity, 3D structural complexity, and imaging conditions. The results of the project will be made available on the Hopkinson lab website (www.hopkinsonlab.org) and code will be made publicly available on Github (https://github.com/bmhopkinson). This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.