OBJECTIVES: WITH "STARE: SPATIOTEMPORAL ADAPTIVE-RESOLUTION ENCODING" WE ADDRESS FOCUS 2.1.3 "CLOUD OPTIMIZED PREPROCESSING AND DATA TRANSFORMATION." CURRENT EARTH SCIENCE DATA PROCESSING FEATURES LARGE CENTRALIZED ARCHIVES PROVIDING EXCEPTIONAL BROWSE AND SEARCH CAPABILITIES USED BY RESEARCHERS WHO IDENTIFY THEN DOWNLOAD DATA IN FILE FORM TO LOCAL COMPUTE/STORAGE RESOURCES FOR PREPROCESSING AND INTEGRATION PRIOR TO ANALYSIS. THIS DATA FLOW FORCES END-USERS TO DEVOTE SCARCE RESOURCES TO SUPPORT THE TRANSFER STORAGE AND MANAGEMENT OF ARCHIVED DATA AS WELL AS SPECIALIST EXPERTISE IN THE VARIOUS DIFFERENT KINDS OF DATA SETS OF THEIR RESEARCH DOMAIN. WE PROPOSE TO SIMPLIFY THIS FLOW BY MOVING PREPROCESSING ACTIVITIES TO THE ARCHIVED DATA ELIMINATING THE COSTS OF TRANSFERRING CREATING AND MAINTAINING REDUNDANT IDIOSYNCRATIC LOCAL ARCHIVES DEVELOPED BY RESEARCHERS WHO ARE GENERALLY NOT ARCHIVISTS NOR THE EXPERT PRODUCERS OF THE ORIGINAL DATA. WITH STARE PROVIDING A UNIFYING PLATFORM FOR DIVERSE DATA MODELS (SWATH POINT GRID) EOSDIS DATA ARCHIVES WILL BE ABLE TO PRODUCE HIGHER-LEVEL PRODUCTS MADE TO ORDER FOR END-USER RESEARCHERS. METHODS: THE CRITICAL NEW TECHNOLOGY WE APPLY IS STARE. STARE'S SPATIAL COMPONENT (SC) HAS DESCENDED FROM THE HIERARCHICAL TRIANGULAR MESH (HTM) SPHERICAL INDEXING ORIGINALLY DEVELOPED FOR THE SLOAN DIGITAL SKY SURVEY IN WHICH STORAGE AND COMPUTATIONAL EFFICIENCY WAS KEY. THE STARE/SC RECURSIVELY DIVIDES THE EARTH'S SURFACE INTO A SET OF QUAD-TREES ALLOWING ANY POINT ON EARTH TO BE IDENTIFIED WITH A SINGLE NUMBER. THE STARE TEMPORAL COMPONENT (TC) HAS SIMILAR PROPERTIES. FOR OBSERVATIONS THESE STARE INDICES CONTAIN BOTH LOCATION AND RESOLUTION INFORMATION PROMOTING EFFICIENT DATA PLACEMENT ON DISTRIBUTED CLOUD RESOURCES MINIMIZING COSTLY DATA TRANSPORT BETWEEN NODES FOR OPERATIONS SUCH AS JOINING INTERSECTING (CONDITIONAL) SUBSETTING AND RE-GRIDDING DIVERSE DATASETS. STARE AUTOMATICALLY CO-ALIGNS DIVERSE DATA IN THE CLOUD PLACING SPATIOTEMPORALLY CLOSE DATA ON THE SAME COMPUTE/STORAGE NODE FOR A RELATIVELY SMALL COST IN METADATA. STARE THUS ALLOWS DIVERSE DATA TO BE EFFICIENTLY INTEGRATED FOR ANALYSIS IN THE CLOUD PROVIDING A FOUNDATION ON WHICH EXISTING TOOLS AND PROCESSING METHODS CAN BE PLACED. MUCH CAPABILITY E.G. PREPROCESSING SEARCHING VISUALIZATION ETC. HAS BEEN DEVELOPED TO SUPPORT RESEARCHERS' USE OF EARTH SCIENCE DATA. IN THE COURSE OF THE PROPOSED WORK WE WILL SHOW HOW EXISTING TOOLS AND METHODS BENEFIT FROM THE STARE-ENABLED PLATFORM. THIS CAN BE VIA A TIGHT INTEGRATION AS HAS BEEN DONE FOR EXAMPLE INCORPORATING STARE WITH THE DISTRIBUTED ARRAY DATABASE SCIDB ALONG WITH RE-GRIDDING FUNCTIONS AND FAST PARALLEL GEOGRAPHIC INTERSECTIONS. OR CURRENT TOOLS MAY SIMPLY BE APPLIED TO THE RESULTS OF STARE-ENABLED DISTRIBUTED PROCESSING E.G. FAST GRANULE INTERSECTION IN A MORE CONVENTIONAL BUT CLOUD-BASED DATA PROCESSING FLOW. SIGNIFICANCE: AS A UNIFYING PLATFORM STARE SUPPORTS CONVENTIONAL PROCESSING ANALYSIS AND VISUALIZATION TOOLS BRINGING THE OPPORTUNITY FOR MASSIVELY INCREASING THE AMOUNT OF DATA RESEARCHERS CAN USE. IN THE LONGER TERM AS TOOLS EVOLVE TO TAKE GREATER ADVANTAGE OF STARE'S INTEGRATIVE CAPABILITIES WE CAN MOVE FROM THE CURRENT FOCUS ON THE EXPENSIVE LOW-LEVEL MANIPULATION OF DATA FILES TO AN ABILITY TO INTERACT WITH EARTH SCIENCE DATA AT A HIGHER LEVEL WITH QUERY-BASED DECLARATIVE TOOLS AND USER INTERFACES THAT FAVOR SCIENTIFIC INQUIRY RATHER THAN DATA MANAGEMENT. AT THE VERY LEAST STARE HELPS AUTOMATE CRITICAL SPATIOTEMPORAL FUNCTIONS WHILE MAKING EFFICIENT USE OF CLOUD COMPUTING PROMISING TO ELIMINATE THE NEED FOR RESEARCHERS TO DEVOTE TIME MONEY AND EXPERTISE TO THE REDUNDANT TRANSFER OF ARCHIVED DATA TO THEIR OWN LOCAL SYSTEMS. THE TIME AND EFFORT SAVED IMPROVES SCIENTIFIC QUALITY AND THE PRODUCTIVITY OF THE CURRENT RESEARCHERS AND REDUCES THE COST-OF-ENTRY FOR OTHERS WHO MIGHT SEEK VALUE FROM EOSDIS DATA RESOURCES.$1,267,396
· FY2020 · National Aeronautics and Space Administration