Hierarchical Data Format for Water-related Big Geodata (HDF4Water)

Project Description

The HDF4Water project is funded by DFG (project no. 460036893) as an 6-month incubator project via the NFDI Consortium Earth System Sciences (NFDI4Earth).

In this project, we developed a big data framework based on the modern HDF5 technology, called AtlasHDF (Werner & Li, 2022), in which we designed lossless data mappings (i.e., immediate mapping and analysis-ready mapping) from OpenStreetMap (OSM) vector data into a single HDF5 data container to facilitate fast and flexible GeoAI applications, for instance, surface water mapping with OSM and Sentinel-2 data. This document describes concisely all decisions that we took towards adapting the flexible AtlasHDF data storage framework to the requirements of a sustainable, simple, (almost) dependency-free data representation for GeoAI across multiple modalities.

Related Resources

• GitHub repository at: https://github.com/tum-bgd/atlashdf
• How to install at: https://github.com/tum-bgd/atlashdf/blob/main/Dockerfile
• How to use at: https://github.com/tum-bgd/atlashdf/blob/main/HDF4Water-Tutorial.ipynb
• Promotion video at: https://youtu.be/An-qhB30CLU

Publications

1. Werner, M., & Li, H. (2022). AtlasHDF: An Efficient Big Data Framework for GeoAI. 1–7. https://doi.org/10.1145/3557917.3567615 [PDF] [Online]