RESEARCHERS IN THE EARTH ATMOSPHERE REMOTE SENSING COMMUNITY NEED TOOLS TO BE ABLE TO EFFECTIVELY HANDLE THE HYPERSPECTRAL DATA THAT IS NOW BECOMING AVAILABLE. THE MASSIVE DATA VOLUMES OF THESE DATASETS DEMANDING UP TO AN ESTIMATED 100 TB OF STORAGE FOR A SET OF ANALYSIS ACTIVITIES TO BE COMPLETED IS A BARRIER TO THEIR USE. WE PROPOSE TO DEVELOP A TOOL HY-LATIS TO FACILITATE THE USE OF HYPERSPECTRAL DATA (HD) BY PROVIDING CONVENIENT ACCESS TO AND OPERATIONS OVER THE MASSIVE DATA VOLUMES THEY PRESENT. THE DATA WILL BE STORED IN THE CLOUD WHERE SERVER-SIDE COMPUTATIONS CAN BE PERFORMED. USERS WILL BE ABLE TO ANALYZE DATA IN THE CLOUD USING A NOTEBOOK APPROACH THAT ENABLES PROCESSING TO OCCUR IN THE CLOUD WITHOUT REQUIRING DATASET DOWNLOAD TO USER WORKSTATIONS. THEY WILL ALSO BE ABLE TO REDUCE HD AND STREAM IT INTO LOCAL MEMORY FOR THE PURPOSE OF INTEGRATING HYPERSPECTRAL INFORMATION INTO EXISTING SEQUENTIAL ANALYSES. OVERALL HD WILL BE EASIER TO ACCESS AND USE WITH BETTER INTERACTIVITY IN ANALYSES THAN IS AVAILABLE TODAY. THE WILL BE ACHIEVED BY PUTTING A FUNCTIONAL DATA PROCESSING LAYER OVER THE SPARK BIG DATA PROCESSING ENGINE LEVERAGING THE EXISTING LATIS DATA ACCESS SERVER AND FRAMEWORK AS THE FUNCTIONAL DATA PROCESSING LAYER. LATIS HAS BEEN IN USE AT THE LABORATORY FOR ATMOSPHERIC AND SPACE PHYSICS (LASP) FOR MANY YEARS WITH ABOUT 20 INSTANCES CURRENTLY RUNNING SERVING A WIDE VARIETY OF DATASETS FROM A WIDE VARIETY OF DOMAINS FORMATS AND LOCATIONS. WE ESTIMATE LATIS TO BE AT AN ENTRY ASSESSMENT OF TRL 4 DUE TO ITS DEMONSTRATED SUCCESS WITHIN THE LAB. THIS PROJECT EXPANDS THE SPARK INTERFACE TO OFFER NOT ONLY THE RELATIONAL DATA MODEL INTERFACE THAT IT DOES NOW BUT ALSO A FUNCTIONAL INTERFACE IN THE FUNCTIONAL PROGRAMMING SENSE. THIS IS ACCOMPLISHED BY IMPLEMENTING THE LATIS FUNCTIONAL DATA MODEL AND ASSOCIATED HYPERSPECTRAL IMAGING ANALYSIS (HIA) OPERATIONS WITH A FUNCTIONAL STYLE WHILE LEVERAGING THE POWER OF SPARK UNDERNEATH. WE ARGUE THAT A FUNCTIONAL APPROACH TOWARDS DATA AND PROCESSING PROVIDES A FLEXIBILITY OF REPRESENTATION COUPLED WITH THE RIGHT SET OF SEMANTICS TO BE A VERY GOOD APPROACH FOR SCIENTIFIC ANALYSIS. THIS TWO-YEAR PROJECT WILL START WITH THE TECHNICAL TEAM WORKING WITH OUR SCIENCE CO-IS WHO ARE OUR SUBJECT MATTER EXPERTS (SMES) TO DEVELOP A CORE HYPERSPECTRAL IMAGE ANALYSIS (HIA) DOMAIN TOOL SET THAT SUPPORTS HD ANALYSIS IN THE CODE AND STREAMING INTO MEMORY REDUCED VOLUME HD. THE TOOL SET WILL PROVIDE AN INTUITIVE INTERFACE DESIGNED SPECIFICALLY AT THE LEVEL OF HIA. THE FIRST OPERATIONS TO IMPLEMENT WILL BE ACCESS OPERATIONS SUCH AS SUBSETTING ON GEOLOCATION WAVELENGTH OR PIXEL. THE SECOND SET OF OPERATIONS WILL BE SERVER SIDE ANALYSES SUCH AS VARIOUS FORMS OF INTEGRATION TO BE APPLIED TO REDUCE THE DATA VOLUME TO A LOWER RESOLUTION. AFTER THIS WORK IS UNDERWAY AND DEPENDING ON THE TIMING OF THE NEEDS OF OUR SMES NEW ANALYSES AND UNDERLYING OPERATIONS WILL BE DEVELOPED THAT LEVERAGE HY-LATIS OPERATIONS VIA NOTEBOOK TECHNOLOGIES SUCH AS JUPYTER NOTEBOOK OR JUPYTERLAB. THIS IS WHERE THE TRUE INTERACTIVE EXPERIENCE WILL LIE. INSTEAD OF THE BATCH ORIENTED HIA PRACTICES OF TODAY THIS APPROACH WILL SAVE TIME BY REDUCING OR ELIMINATING THE TEDIOUS DOWNLOAD-EXTRACT-OPERATE PROCESSING NOW REQUIRED. ONCE THE CAPABILITIES OF SUBSETTING INTERPOLATING AND PERFORMING A FEW OTHER ESSENTIAL OPERATIONS ARE OPERATIONAL IN A NOTEBOOK ENVIRONMENT THE HY-LATIS DEVELOPMENT TEAM WILL HOLD A WORKSHOP TO DEMONSTRATE THE TOOL TO THE EARTH ATMOSPHERE REMOTE SENSING COMMUNITY. THERE THEY WILL BE ABLE TO TEST THE TOOL AND PROVIDE FEEDBACK. OUR SUCCESS WILL BE DETERMINED BY THE ADOPTION AND USAGE OF THE TOOLS BY OUR SMES AND IN PERFORMING SCIENCE RESEARCH OPERATIONS THAT INVOLVE THESE HYPERSPECTRAL DATASETS.
$1,069,973FY2020National Aeronautics and Space AdministrationNASA
The Regents Of The University Of Colorado