HADOOP 3.0.0 之 HDFS RAID

author: fenqi@mogujie
date: 2016/05/19

背景

  • 场景1:实时访问冷数据存储
  • 场景2:归档数据库备份

    目标:降低存储成本,降低扩容难度

选型

现有方案

  1. Apache官方方案
    Hadoop with Erasure Code inside: 1, 2. 计划在Apcache Hadoop 3.0中发布,目前还未release,代码已经合并到trunk,java版本要求是JDK 1.8以上。

    1
    2
    3
    4
    5
    Requirements:
    * Unix System
    * JDK 1.8+
    * Maven 3.0 or later

由于目前是alpha版本,所以没有使用案例

  • Facebook方案
    HDFS RAID introduced by Facebook,github上的代码14年开始不再维护了,这个方案存在一些问题,对于我们需要使用的场景有影响的是:

    Secondly, when files are replicated across clusters, sometimes parity files are not moved together with their source files. This often leads to data loss.

    有两种编码:XOR and Reed-Solomon
    国内公司使用情况了解到的:豌豆荚在使用;腾讯从2014年开始使用,目前仍在大规模使用;之前淘宝使用过,但是云梯集群目前已下线。
    由于HDFS RAID是基于hadoop 0.20版本上做的更改,而且方案在一些场景下有问题,所以使用基本上都需要移植

  • USC方案
    Xorbas, from USC paper working with facebook. github上代码12年开始没有更新

  • 其它:usenix paper

初步倾向

选型关注点:方案的可重用性、可持续性,团队成长,项目周期

  1. 在hadoop稳定版本上移植HDFS RAID的功能,开发周期长,目前没有相关积累,且后面可能会面临升级到Hadoop3的问题;
  • 方案重用的意义不大,虽然有一些价值,但这个时间点和Hadoop3发布时间点太近;
  • 目前业务需求不是特别紧急,可以在hadoop 3.0.0上多做一些测试验证
  • 在hadoop 3.0.0的基础上做开发维护,方案更有可持续性,经验也可以重用到其它现有集群

    倾向于谨慎使用hadoop 3.0.0

Hadoop 3.0.0 安装

系统环境

1
2
vps: bandwagon (Los Angeles)
os: CentOS release 6.6 (Final)

软件版本

1
2
3
Repos: git://git.apache.org/hadoop.git
Revision: 1597630681c784a3d59f5605b87e96197b8139d7
Date: Wed May 18 18:46:00 2016 -0700

安装过程

  1. clone source

    1
    git clone git://git.apache.org/hadoop.git
  2. install java8

  3. install packages

    1
    sudo yum install autoconf automake libtool cmake zlib-devel gcc-c++ openssl-devel
  4. install protobuf2.5.0

  5. 编译

    1
    mvn package -Pdist,native -DskipTests -Dtar
  6. 完成

    1
    Hadoop dist tar available at: /home/fenqi/code/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha1-SNAPSHOT.tar.gz

部署

同hadoop-2.6.0-cdh5.7.0相同方式部署,namenode web ui端口从之前默认的50070变为9870

1
2
3
4
5
6
7
<property>
<name>dfs.namenode.http-address</name>
<value>0.0.0.0:9870</value>
<description>
The address and the base port where the dfs namenode web ui will listen on.
</description>
</property>

从web上可以更改文件所属的用户、组和权限,但是操作会报错

1
2
$ hdfs erasurecode -setPolicy /
EC policy set successfully at hdfs://mogunode:9000/
1
2
$ hdfs erasurecode -getPolicy /
ErasureCodingPolicy=[Name=RS-DEFAULT-6-3-64k, Schema=[ECSchema=[Codec=rs-default, numDataUnits=6, numParityUnits=3]], CellSize=65536 ]
1
2
$ hdfs erasurecode -listPolicies
RS-DEFAULT-3-2-64k, RS-DEFAULT-6-3-64k, RS-LEGACY-6-3-64k

尝试用之前安装的hadoop-hdfs-fuse-2.6.0+cdh5.7.0写入文件,写入到128M就会报错:

1
2
3
4
5
$ dd if=/dev/zero bs=4096 count=1048576 of=file_4GB
dd: 正在写入"file_4GB": 输入/输出错误
记录了66823+0 的读入
记录了66822+0 的写出
273702912字节(274 MB)已复制,16.4314 秒,16.7 MB/秒

namenode上无ERROR级别日志,只有INFO级别异常:

1
2
3
4
org.apache.hadoop.hdfs.server.namenode.NotReplicatedYetException: Not replicated yet: /file_4GB
at org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.validateAddBlock(FSDirWriteFileOp.java:200)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2484)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:844)

使用hadoop shell -copyFromLocal 写入,可以看到错误原因:

1
2
$ hadoop fs -copyFromLocal file_4GB /file_4GB
copyFromLocal: Failed to get 6 nodes from namenode: blockGroupSize= 9, blocks.length= 4

启动了7个节点之后,写入成功,但是有很多WARN级别日志:

1
2
3
4
5
6
7
8
9
10
11
12
$ hadoop fs -copyFromLocal file_4GB /file_4GB
16/05/19 20:12:08 WARN hdfs.DFSOutputStream: Failed to get block location for parity block, index=7
16/05/19 20:12:08 WARN hdfs.DFSOutputStream: Failed to get block location for parity block, index=8
16/05/19 20:12:34 INFO hdfs.DFSOutputStream: replacing previously failed streamer #7: failed, block==null
16/05/19 20:12:34 INFO hdfs.DFSOutputStream: replacing previously failed streamer #8: failed, block==null
……repeat……
16/05/19 20:14:02 WARN hdfs.DFSOutputStream: Block group <1> has 2 corrupt blocks.
16/05/19 20:14:02 WARN hdfs.DFSOutputStream: Block group <2> has 2 corrupt blocks.
16/05/19 20:14:02 WARN hdfs.DFSOutputStream: Block group <3> has 2 corrupt blocks.
16/05/19 20:14:02 WARN hdfs.DFSOutputStream: Block group <4> has 2 corrupt blocks.
16/05/19 20:14:02 WARN hdfs.DFSOutputStream: Block group <5> has 2 corrupt blocks.
16/05/19 20:14:02 WARN hdfs.DFSOutputStream: Block group <6> has 2 corrupt blocks.

最终:

1
DFS Used: 4.7 GB (0.12%)

副本因子变为:1.175 (RS-DEFAULT-6-3-64k的理论值是1.5,这里和预期不一致)

3副本的情况下写入之后占用空间是12GB

1
DFS Used: 12.09 GB (0.32%)

致谢

@涂旭东
@吴镝
@张友东
@姜博
@腾讯大数据

引用

[1]http://www.cnblogs.com/foxmailed/p/4336266.html
[2]http://jiangbo.me/blog/2013/06/05/setup-hdfs-raid/
[3]http://data.qq.com/article?id=815
[4]http://data.qq.com/article?id=2929
[5]https://wiki.apache.org/hadoop/HDFS-RAID
[6]https://www.usenix.org/node/188447
[7]http://www.csdn.net/article/2013-07-04/2816109-Hadoop_with_XOR_Erasure_Code_spare_bandwidth
[8]https://blog.cloudera.com/blog/2015/09/introduction-to-hdfs-erasure-coding-in-apache-hadoop/
[9]http://blog.cloudera.com/blog/2016/02/progress-report-bringing-erasure-coding-to-apache-hadoop/
[10]http://smahesh.com/HadoopUSC/
[11]https://code.facebook.com/posts/536638663113101/saving-capacity-with-hdfs-raid/