栅格地理数据并行格式转换引擎

如今大规模地理数据正在社会各个部门和组织中迅速积累,但是由于部门利益和历史沿袭等原因,大规模地理数据共享仍然极具挑战,相应共享技术需求仍然极其旺盛。作为地理数据共享的基础方式之一,传统单机地理数据格式转换技术,一方面受限于磁盘读写及带宽瓶颈,另一方面面对日趋庞大的数据规模,已很难满足实际应用需求。因此提出一种针对栅格地理数据的并行格式转换引擎,采用高性能计算集群环境支持大规模栅格地理数据转换共享,大幅降低了大规模栅格地理数据转换过程的时间成本。栅格地理数据并行格式转换引擎采用基于公共接口的设计理念,框架灵活、具有良好的扩展性,支持地理数据格式的读写自定义以及新数据格式添加,能够实现接入数据格式间的任意两两转换。为验证引擎框架及其处理效率,在Lustre并行集群环境下以格网数据交换格式(国家地理空间数据交换格式)向常见栅格地理格式的转换为示例进行了测试实验。结果表明,栅格地理数据并行格式转换引擎能够在8个节点Lustre集群中达到7.54的良好并行加速比。

Parallel geo-raster data conversion engine

Large scale of geo-Raster data have been accumulated all over the world during the past decades, but quite often in various data formats, resulting in geospatial data sharing as an everlasting headache to the GIS world. However, despite of a variety of methodologies created in the past, geospatial data conversion has always been a fundamental and efficient way enabling geospatial data sharing, but nowadays as the size of data tends to be larger and larger the methodology need a re-write. This paper proposed a Parallel Geo-Raster data Conversion Engine (PGRCE) to achieve massive Geo-Raster data efficient sharing by utilizing high performance computing technology. PGRCE is designed in an extendable and flexible framework, and is capable of customizing the way of reading and writing for particular spatial data formats. An experiment, in which geo-Raster data in the CNSDTF-DEM format (Raster spatial data defined in Chinese Geospatial Data Transfer Format Standard) were transferred using PGRCE in a parallel file system (Lustre), are conducted to validate the engine framework and its performance. Results show that PGRCE could achieve a 7.54 speedup on a Luster cluster of 8 nodes.