| 分布式计算环境下的栅格数据存储策略 |
| |
| 引用本文： | 张剑波,夏灯城,赵加奥,李谢清,崔永键,袁国斌.分布式计算环境下的栅格数据存储策略[J].国防科技大学学报,2017,39(6):51-58. |
| |
| 作者姓名： | 张剑波 夏灯城 赵加奥 李谢清 崔永键 袁国斌 |
| |
| 作者单位： | 中国地质大学武汉 信息工程学院,中国地质大学武汉 信息工程学院,中国地质大学武汉 信息工程学院,中国地质大学武汉 信息工程学院,中国地质大学武汉 信息工程学院,中国地质大学武汉 信息工程学院 |
| |
| 基金项目： | 国家自然科学基金项目（41001225，41501584） |
| |
| 摘 要： | 针对传统的栅格数据存储策略不能满足分布式计算环境下粗粒度数据访问需求,应对海量栅格数据计算时效率低下的问题,结合分布式文件系统的存储特点,同时考虑地图代数算子在Map/Reduce阶段以栅格瓦片为单位的计算特点,提出一种基于Hadoop分布式文件系统的栅格瓦片存储策略。围绕栅格数据瓦片分割、压缩瓦片数据组织与存储、分布式文件输入输出接口改进等方面对该存储策略加以实现,并使用基于该存储策略的地图代数局部算子的分布式计算流程加以验证。理论分析与实验结果表明,该策略能够显著提高分布式计算环境下空间分析算子的运算速度。
|
| 关 键 词： | 分布式计算 栅格数据 存储策略 地图代数 |
| 收稿时间： | 2016/7/13 0:00:00 |
| 修稿时间： | 2017/1/15 0:00:00 |
Storage StrategySof Raster Data under the DistributedSComputing Environment |
| |
| ZHANG Jianbo,XIA Dengcheng,ZHAO Jiaao,LI Xieqing,CUI Yongjian and YUAN Guobin.Storage StrategySof Raster Data under the DistributedSComputing Environment[J].Journal of National University of Defense Technology,2017,39(6):51-58. |
| |
| Authors: | ZHANG Jianbo XIA Dengcheng ZHAO Jiaao LI Xieqing CUI Yongjian and YUAN Guobin |
| |
| Institution: | Faculty of Information Engineering, China University of Geosciences, Wuhan 430074, China,Faculty of Information Engineering, China University of Geosciences, Wuhan 430074, China,Faculty of Information Engineering, China University of Geosciences, Wuhan 430074, China,Faculty of Information Engineering, China University of Geosciences, Wuhan 430074, China,Faculty of Information Engineering, China University of Geosciences, Wuhan 430074, China and Faculty of Information Engineering, China University of Geosciences, Wuhan 430074, China |
| |
| Abstract: | Traditional storage strategy of raster data cannot meet the demands of coarse-grained data processing under the distributed computing environment and has low efficiency when dealing with calculations for gigantic raster data. A storage strategy of raster tile data is presented on the basis of the storage characteristics of distributed file system. It also takes the calculation characteristics of spatial analysis operators of map algebra into consideration, which use raster tile as processing unit during map and reduce stage. The storage strategy is implemented by following steps. Firstly raster data are divided into raster tiles. Then these tiles are compressed and organized by a special sequence in order to transferring to Hadoop distributed file system. Finally input and output file interfaces are re-implemented to meet data access requirements of map and reduce stage. The strategy is tested and verified by distributed calculation process of local map algebra operators. Theoretical analysis and experimental results show that this strategy can significantly improve the processing speed of space analysis operators. |
| |
| Keywords: | distributed computing raster data storage strategy map algebra |
| 本文献已被 CNKI 等数据库收录！ |
| 点击此处可从《国防科技大学学报》浏览原始摘要信息 |
| 点击此处可从《国防科技大学学报》下载免费的PDF全文 |
|
