Dynamic Data Placement Strategy in MapReduce-styled Data Processing Platform

Abstract

Data placement is one of the core technologies of the MapReduce-styled data processing platform, which contributes much to the data processing efficiency. A dynamic data placement strategy is proposed in this paper due to that existing data placement techniques are lack of the consideration of the computing load on the data storage node and reduce the ratio of the localized processing of the hot-spot data. The strategy takes the data accessing localization ratio, the remaining computing capacity of nodes as the new factors of the data placement decision. Performance evaluation results show that the proposed data placement strategy outperforms the original strategy in HDFS file system and the average job execution time is reduce by the maximum of 12%.