Large Data
WASP
Project Description
Advances in digital imaging methods are facilitating the acquisition of huge amounts of data that allow the visualization and analysis of complex, multi-dimensional images. At the same time, modern computing technologies enable numerical modeling of a broad array of scientific phenomena, resulting in vast quantities of numerical data. The analysis and visualization of these types of large data play an important role in scientific discovery across many different fields and require efficient and broadly available tools to accomplish this task. However, efficient management of large data for analysis and visualization is a non-trivial problem as the size and complexity of data increases. We have been working to address this challenging problem through development of a common software framework that uses a general progressive access, multi-scale data refinement method for efficient handling of very large scientific data sets. This approach, that we call Wavelet-enabled progressive data Access and Storage Protocol (WASP), is based upon an underlying wavelet-based data representation developed by NCAR for geoscience applications. The large data visualization tools that we have developed utilize the very flexible and open source standard NetCDF format and includes a documented set of conventions and a toolkit that integrates these components for dissemination. In addition, these tools have been integrated into the VAPOR and QUEST software, thus expanding the capabilities and efficiencies of these applications for the geo- and bio- sciences communities. The development of this general toolkit for wavelet-based representations of data facilitates the multi-scale analysis, storage, and visualization for data collected in a wide range of fields and on a multitude of applications, from high-end computing facilities to laptop computers used by students, field biologists, and others. Funding provided by NSF ACI-1440412: SI2-SSE: Wavelet Enabled Progressive Data Access and Storage Protocol (WASP). Principal Investigators: Dr. Lawrence Frank and John Clyne (NCAR).
See the WASP project website waspdata.org for more details.
Publications
Frank, L.R. and Clyne, J. (2016). SI2-SSE: Wavelet-enabled progressive data Access and Storage Protocol (WASP). NSF SI2 PI Workshop, Arlington, VA. (Available at: waspdata.org)
Clyne J, Frank L, Lesperance T, Norton, A, Pearse S. WASP: Intelligent Storage for Gridded Numerical Data (Poster). 5th Annual Extreme Science and Engineering Discovery Environment Conference (XSEDE16). Miami, FL. (July 19, 2016) (Available at: waspdata.org)