Hadoop2.7.3高可用(HA)集群部署

一.部署前准备工作
VMWare

三台虚拟机:Centos6.5 x64 操作系统(需自行安装,先安装一台,然后克隆两台)

Hadoop 2.7.3 64位安装包

JDK 1.8

Zookeeper 3.4.9 安装包


VV整理好了本篇文章所用的软件包。

扫描下方二维码,关注此公众号,关注后,输入“ hadoop集群 ”即可获取哦~

想要更多大数据学习资料吗?赶快关注吧

回复“ 资料 ”可获取大数据学习资料,分享到朋友圈更是有额外的课件、资料赠送哦!



二.集群规划
主机名
IP地址
安装的软件
运行的进程
louisvv01 (主节点)
192.168.1.210
JDK、Hadoop
NameNode、ResourceManager、QuorumPeerMain
DataNode、DFSZKFailoverController、NodeManager、JournalNode
louisvv02
192.168.1.211
JDK、Hadoop
NameNode(StandBy)、ResourceManager(StandBy)、QuorumPeerMain
DataNode、DFSZKFailoverController、NodeManager、JournalNode
louisvv03
192.168.1.212
JDK、Hadoop
QuorumPeerMain、DataNode、NodeManager、JournalNode
三.基础环境

1.IP地址(需要在三台机器上都进行配置)

2.修改主机名,根据集群规划来进行修改(需要在三台机器上都进行配置)
临时修改主机名
[root@louisvv ~]# hostname louisvv01
[root@louisvv ~]# hostname
louisvv01
永久修改主机名
[root@louisvv ~]# vim /etc/sysconfig/network
NETWORKING=yes
HOSTNAME=louisvv01
3.修改主机名和IP的映射关系/etc/hosts(需要在三台机器上都进行配置)
[root@louisvv ~]# vim /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.1.210 louisvv01
192.168.1.211 louisvv02
192.168.1.212 louisvv03
4.关闭防火墙,关闭防火墙开启自启(需要在三台机器上都进行配置)
[root@louisvv ~]# service iptables stop
[root@louisvv ~]# chkconfig iptables off
5.ssh免登陆(在louisvv01上进行配置)
[root@louisvv01 ~]# ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
2e:75:72:ff:54:65:51:e7:e6:cb:42:e9:45:3e:ac:7f root@louisvv.com
The key’s randomart image is:
+–[ RSA 2048]—-+
| .+|
| .o|
| .=|
| =+.|
| S o o =o|
| o + .o +.o|
| . . .+.o |
| . oo E|
| …|
+—————–+
执行完上面的命令后,会生成两个文件id_rsa(私钥)、id_rsa.pub(公钥),将公钥拷贝到要免登陆的机器上
[root@louisvv ~]# ssh-copy-id louisvv02
[root@louisvv ~]# ssh-copy-id louisvv03
测试一下,ssh免登陆
[root@louisvv ~]# ssh louisvv02
Last login: Wed Dec 6 10:27:15 2017 from 192.168.1.74
[root@louisvv02 ~]#
[root@louisvv ~]# ssh louisvv03
Last login: Wed Dec 6 10:27:20 2017 from 192.168.1.74
6.安装JDK
使用winscp软件将JDK包上传到louisvv01机器上/opt目录下
[root@louisvv ~]# cd /opt/
[root@louisvv opt]# ls
jdk-8u91-linux-x64.tar.gz
[root@louisvv opt]# tar -zxf jdk-8u91-linux-x64.tar.gz
[root@louisvv opt]# ls
jdk1.8.0_91 jdk-8u91-linux-x64.tar.gz
配置JDK环境变量
[root@louisvv opt]# vim /etc/profile
#java env
export JAVA_HOME=/opt/jdk1.8.0_91
export JRE_HOME=/opt/jdk1.8.0_91/jre
export CLASSPATH=$JAVA_HOME/lib
export PATH=:$PATH:$JAVA_HOME/bin:$JRE_HOME/bin:
刷新环境变量,并测试java环境变量配置是否成功
[root@louisvv opt]# source /etc/profile
[root@louisvv opt]# java -version
java version “1.8.0_91”
Java(TM) SE Runtime Environment (build 1.8.0_91-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.91-b14, mixed mode)
配置好JDK后,将配置好的JDK和/etc/profile远程拷贝到另外两台机器上
[root@louisvv opt]# scp -r jdk1.8.0_91/ louisvv02:/opt/
[root@louisvv opt]# scp -r jdk1.8.0_91/ louisvv03:/opt/
[root@louisvv opt]# scp /etc/profile louisvv02:/etc/
[root@louisvv opt]# scp /etc/profile louisvv03:/etc/
在louisvv02,louivv03上刷新环境变量,并测试java环境变量是否配置成功
[root@louisvv ~]# source /etc/profile
[root@louisvv ~]# java -version
java version “1.8.0_91”
Java(TM) SE Runtime Environment (build 1.8.0_91-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.91-b14, mixed mode)
四.Zookeeper集群安装
配置好基础环境后,要进行Zookeeper集群的安装
1.使用winscp软件,将zookeeper-3.4.9.tar.gz安装包上传到/opt目录下
2.解压
[root@louisvv opt]# tar -zxf zookeeper-3.4.9.tar.gz
3.配置zookeeper
[root@louisvv opt]# cd zookeeper-3.4.9
[root@louisvv zookeeper-3.4.9]# cd conf/
[root@louisvv conf]# ls
configuration.xsl log4j.properties zoo_sample.cfg
[root@louisvv conf]# cp zoo_sample.cfg zoo.cfg
修改配置文件
[root@louisvv conf]# vim zoo.cfg
修改data目录地址
dataDir=/opt/zookeeper-3.4.9/data/
并在文件最后添加
server.1=louisvv01:2888:3888
server.2=louisvv02:2888:3888
server.3=louisvv03:2888:3888
创建相应的data dir目录和myid文件,其中,这个myid文件必须创建,否则启动会报错
myid一定对应好zoo.cfg中配置的server后面1、2、3
[root@louisvv conf]# mkdir /opt/zookeeper-3.4.9/data
[root@louisvv data]# vim /opt/zookeeper-3.4.9/data/myid
1
将配置好的zookeeper scp拷贝到另外两台机器
[root@louisvv opt]# scp -r /opt/zookeeper-3.4.9 louisvv02:/opt/
[root@louisvv opt]# scp -r /opt/zookeeper-3.4.9 louisvv03:/opt/
在louisvv02、louisvv03上分别修改 /opt/zookeeper-3.4.9/data/myid 文件,改为对应的id号
louisvv02为2,louisvv03为3
4.添加环境变量(在louisvv02、louisvv03上也进行配置)
[root@louisvv zookeeper-3.4.9]# vim /etc/profile
#zookeeper env
export ZOOKEEPER_HOME=/opt/zookeeper-3.4.9
#hadoop env
export HADOOP_HOME=/opt/hadoop-2.7.3
export PATH=:$PATH:$JAVA_HOME/bin:$JRE_HOME/bin:$ZOOKEEPER_HOME/bin:
保存退出,刷新环境变量
[root@louisvv zookeeper-3.4.9]# source /etc/profile
[root@louisvv zookeeper-3.4.9]# zk
zkCleanup.sh zkCli.cmd zkCli.sh zkEnv.cmd zkEnv.sh zkServer.cmd zkServer.sh
5.开启zookeeper(在三台机器上执行)
[root@louisvv zookeeper-3.4.9]# zkServer.sh start
ZooKeeper JMX enabled by default
Using config: /opt/zookeeper-3.4.9/bin/../conf/zoo.cfg
Starting zookeeper … STARTED
使用jps命令查看zookeeper对应java进程,发现QuorumPeerMain正在运行,即zookeeper开启成功
[root@louisvv zookeeper-3.4.9]# jps
2713 Jps
2671 QuorumPeerMain
检查zookeeper集群是否启动成功,发现三台机器上,有两台是follower,一台是leader,即zookeeper安装成功,并成功即启动
[root@louisvv zookeeper-3.4.9]# zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /opt/zookeeper-3.4.9/bin/../conf/zoo.cfg
Mode: follower
[root@louisvv03 ~]# zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /opt/zookeeper-3.4.9/bin/../conf/zoo.cfg
Mode: leader
[root@louisvv02 ~]# zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /opt/zookeeper-3.4.9/bin/../conf/zoo.cfg
Mode: follower
五.安装hadoop
使用winscp软件将hadoop-2.7.3.tar.gz包上传到louisvv01机器的 /opt目录下
1.解压
[root@louisvv opt]# tar -zxf hadoop-2.7.3.tar.gz
2.进入解压后目录,修改配置文件
[root@louisvv opt]# cd /opt/hadoop-2.7.3/etc/hadoop/
先修改hadoop-env.sh修改JAVA HOME
[root@louisvv hadoop]# vim hadoop-env.sh
export JAVA_HOME=/opt/jdk1.8.0_91
修改core-site.xml添加如下内容:
<configuration>
<!– 指定hdfs的nameservice为ns1 –>
<property>
<name>fs.defaultFS</name>
<value>hdfs://ns1</value>
</property>
<!– 指定hadoop临时目录 –>
<property>
<name>hadoop.tmp.dir</name>
<value>/opt/hadoop-2.7.3/tmp</value>
</property>
<!– 指定zookeeper地址 –>
<property>
<name>ha.zookeeper.quorum</name>
<value>louisvv01:2181,louisvv02:2181,louisvv03:2181</value>
</property>
</configuration>
修改hdfs-site.xml添加如下内容:
<configuration>
<!–指定hdfs的nameservice为ns1,需要和core-site.xml中的保持一致 –>
<property>
<name>dfs.nameservices</name>
<value>ns1</value>
</property>
<!– ns1下面有两个NameNode,分别是nn1,nn2 –>
<property>
<name>dfs.ha.namenodes.ns1</name>
<value>nn1,nn2</value>
</property>
<!– nn1的RPC通信地址 –>
<property>
<name>dfs.namenode.rpc-address.ns1.nn1</name>
<value>louisvv01:9000</value>
</property>
<!– nn1的http通信地址 –>
<property>
<name>dfs.namenode.http-address.ns1.nn1</name>
<value>louisvv01:50070</value>
</property>
<!– nn2的RPC通信地址 –>
<property>
<name>dfs.namenode.rpc-address.ns1.nn2</name>
<value>louisvv02:9000</value>
</property>
<!– nn2的http通信地址 –>
<property>
<name>dfs.namenode.http-address.ns1.nn2</name>
<value>louisvv02:50070</value>
</property>
<!– 指定NameNode的元数据在JournalNode上的存放位置 –>
<property>
<name>dfs.namenode.shared.edits.dir</name>
<value>qjournal://louisvv01:8485;louisvv02:8485;louisvv03:8485/ns1</value>
</property>
<!– 指定JournalNode在本地磁盘存放数据的位置 –>
<property>
<name>dfs.journalnode.edits.dir</name>
<value>/opt/hadoop-2.7.3/journal</value>
</property>
<!– 开启NameNode失败自动切换 –>
<property>
<name>dfs.ha.automatic-failover.enabled</name>
<value>true</value>
</property>
<!– 配置失败自动切换实现方式 –>
<property>
<name>dfs.client.failover.proxy.provider.ns1</name>
<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
</property>
<!– 配置隔离机制方法,多个机制用换行分割,即每个机制暂用一行–>
<property>
<name>dfs.ha.fencing.methods</name>
<value>
sshfence
shell(/bin/true)
</value>
</property>
<!– 使用sshfence隔离机制时需要ssh免登陆 –>
<property>
<name>dfs.ha.fencing.ssh.private-key-files</name>
<value>/home/hadoop/.ssh/id_rsa</value>
</property>
<!– 配置sshfence隔离机制超时时间 –>
<property>
<name>dfs.ha.fencing.ssh.connect-timeout</name>
<value>30000</value>
</property>
</configuration>
复制mapred-site.xml实例文件
[root@louisvv hadoop]# cp mapred-site.xml.template mapred-site.xml
修改mapred-site.xml文件,添加如下内容:
<configuration>
<!– 指定mr框架为yarn方式 –>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>
修改yarn-site.xml文件,添加如下内容:
<configuration>
<!– 开启RM高可靠 –>
<property>
<name>yarn.resourcemanager.ha.enabled</name>
<value>true</value>
</property>
<!– 指定RM的cluster id –>
<property>
<name>yarn.resourcemanager.cluster-id</name>
<value>yrc</value>
</property>
<!– 指定RM的名字 –>
<property>
<name>yarn.resourcemanager.ha.rm-ids</name>
<value>rm1,rm2</value>
</property>
<!– 分别指定RM的地址 –>
<property>
<name>yarn.resourcemanager.hostname.rm1</name>
<value>louisvv01</value>
</property>
<property>
<name>yarn.resourcemanager.hostname.rm2</name>
<value>louisvv02</value>
</property>
<!– 指定zk集群地址 –>
<property>
<name>yarn.resourcemanager.zk-address</name>
<value>louisvv01:2181,louisvv02:2181,louisvv03:2181</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
</configuration>
修改slaves文件,添加如下内容:
[root@louisvv hadoop]# vim slaves
louisvv01
louisvv02
louisvv03
将配置好的hadoop拷贝到另外两个节点
[root@louisvv opt]# scp -r /opt/hadoop-2.7.3 louisvv02:/opt/
[root@louisvv opt]# scp -r /opt/hadoop-2.7.3 louisvv03:/opt/
将hadoop添加到环境变量(三台机器上全部操作)
export HADOOP_HOME=/opt/hadoop-2.7.3

export PATH=:$PATH:$JAVA_HOME/bin:$JRE_HOME/bin:

$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$ZOOKEEPER_HOME/bin:

六:开启hadoop集群
1.先开启journalnode(在主节点louisvv01上执行)
[root@louisvv opt]# hadoop-daemons.sh start journalnode
louisvv01: starting journalnode, logging to /opt/hadoop-2.7.3/logs/hadoop-root-journalnode-louisvv01.out
louisvv02: starting journalnode, logging to /opt/hadoop-2.7.3/logs/hadoop-root-journalnode-louisvv02.out
louisvv03: starting journalnode, logging to /opt/hadoop-2.7.3/logs/hadoop-root-journalnode-louisvv03.out
jps查看相应java进程,查看是否三台机器都起来了
[root@louisvv opt]# jps
3162 JournalNode
3211 Jps
2671 QuorumPeerMain
2.格式化hdfs,在主节点louisvv01上执行
[root@louisvv opt]# hdfs namenode -format
17/12/06 14:18:12 INFO common.Storage: Storage directory /opt/hadoop-2.7.3/tmp/dfs/name has been successfully formatted.
格式化后会在根据core-site.xml中的hadoop.tmp.dir配置生成个文件
这里我配置的是 /opt/hadoop-2.7.3/tmp/,然后将 /opt/hadoop-2.7.3/tmp/拷贝到 /opt/hadoop-2.7.3/目录下。
[root@louisvv hadoop-2.7.3]# scp -r /opt/hadoop-2.7.3/tmp/ louisvv02:/opt/hadoop-2.7.3/
3.格式化zk,在louisvv01上执行
[root@louisvv hadoop-2.7.3]# hdfs zkfc -formatZK
17/12/06 15:05:32 INFO ha.ActiveStandbyElector: Successfully created /hadoop-ha/ns1 in ZK.
4.开启hdfs
[root@louisvv hadoop-2.7.3]# start-dfs.sh
Starting namenodes on [louisvv01 louisvv02]
louisvv02: starting namenode, logging to /opt/hadoop-2.7.3/logs/hadoop-root-namenode-louisvv02.out
louisvv01: starting namenode, logging to /opt/hadoop-2.7.3/logs/hadoop-root-namenode-louisvv01.out
louisvv03: starting datanode, logging to /opt/hadoop-2.7.3/logs/hadoop-root-datanode-louisvv03.out
louisvv01: starting datanode, logging to /opt/hadoop-2.7.3/logs/hadoop-root-datanode-louisvv01.out
louisvv02: starting datanode, logging to /opt/hadoop-2.7.3/logs/hadoop-root-datanode-louisvv02.out
Starting journal nodes [louisvv01 louisvv02 louisvv03]
由于我们前面开启了journalnode,所以这里显示journalnode已运行
louisvv02: journalnode running as process 3259. Stop it first.
louisvv01: journalnode running as process 3162. Stop it first.
louisvv03: journalnode running as process 3273. Stop it first.
Starting ZK Failover Controllers on NN hosts [louisvv01 louisvv02]
louisvv01: starting zkfc, logging to /opt/hadoop-2.7.3/logs/hadoop-root-zkfc-louisvv01.out
louisvv02: starting zkfc, logging to /opt/hadoop-2.7.3/logs/hadoop-root-zkfc-louisvv02.out
jps命令查看进程是否启动
louisvv01:
[root@louisvv01 ~]# jps
3954 Jps
3527 DataNode
3162 JournalNode
3434 NameNode
3838 DFSZKFailoverController
2671 QuorumPeerMain
louisvv02:
[root@louisvv02 ~]# jps
3466 DataNode
2939 QuorumPeerMain
3259 JournalNode
3612 DFSZKFailoverController
3727 Jps
3407 NameNode
louisvv03:
[root@louisvv03 ~]# jps
3520 Jps
3381 DataNode
2937 QuorumPeerMain
3273 JournalNode
5.开启yarn
[root@louisvv01 ~]# start-yarn.sh
starting yarn daemons
starting resourcemanager, logging to /opt/hadoop-2.7.3/logs/yarn-root-resourcemanager-louisvv01.out
louisvv02: starting nodemanager, logging to /opt/hadoop-2.7.3/logs/yarn-root-nodemanager-louisvv02.out
louisvv03: starting nodemanager, logging to /opt/hadoop-2.7.3/logs/yarn-root-nodemanager-louisvv03.out
louisvv01: starting nodemanager, logging to /opt/hadoop-2.7.3/logs/yarn-root-nodemanager-louisvv01.out
开启ResourceManager备节点
在louisvv02上执行:
[root@louisvv02 ~]# yarn-daemon.sh start resourcemanager
starting resourcemanager, logging to /opt/hadoop-2.7.3/logs/yarn-root-resourcemanager-louisvv02.out
6.查看jps
louisvv01:
[root@louisvv01 ~]# jps
4100 NodeManager
4005 ResourceManager
3527 DataNode
3162 JournalNode
3434 NameNode
4396 Jps
3838 DFSZKFailoverController
2671 QuorumPeerMain
louisvv02:
[root@louisvv02 ~]# jps
3892 ResourceManager
3770 NodeManager
3466 DataNode
2939 QuorumPeerMain
3259 JournalNode
3612 DFSZKFailoverController
3967 Jps
3407 NameNode
louisvv03:
[root@louisvv03 ~]# jps
3554 NodeManager
3381 DataNode
3654 Jps
2937 QuorumPeerMain
3273 JournalNode
7.在web页面上查看hdfs
一个Active,一个Standby,hdfs HA成功

8.在web页面上查看yarn

在网页上输入http://192.168.1.210:8088/cluster/cluster,ResourceManger状态为Active


在网页上输入http://192.168.1.211:8088/cluster/cluster,ResourceManger状态为StandBy


ResourceManger HA成功!
9.测试HDFS的HA
首先向HDFS上传一个文件
[root@louisvv01 ~]# cat test.txt
hello world
hello you
hello me
hello louisvv
[root@louisvv01 ~]# hadoop fs -put test.txt /
[root@louisvv01 ~]# hadoop fs -ls /
Found 1 items
-rw-r–r– 3 root supergroup 45 2017-12-06 15:25 /test.txt
然后我们杀掉louisvv01上那个active的Namenode
[root@louisvv01 ~]# jps
4100 NodeManager
4005 ResourceManager
3527 DataNode
3162 JournalNode
3434 NameNode
4539 Jps
3838 DFSZKFailoverController
2671 QuorumPeerMain
[root@louisvv01 ~]# kill -9 3434
[root@louisvv01 ~]# jps
4100 NodeManager
4005 ResourceManager
4549 Jps
3527 DataNode
3162 JournalNode
3838 DFSZKFailoverController
2671 QuorumPeerMain
已经将Namenode的进程杀死
去louisvv02 hdfs web页面查看其状态是否切换

发现其状态为Active

再查看一下刚才上传的文件,刚才上传的文件也在
[root@louisvv01 ~]# hadoop fs -ls /
Found 1 items
-rw-r–r– 3 root supergroup 45 2017-12-06 15:25 /test.txt
将刚才杀掉的Namenode再启动起来
[root@louisvv01 ~]# hadoop-daemon.sh start namenode
starting namenode, logging to /opt/hadoop-2.7.3/logs/hadoop-root-namenode-louisvv01.out

通过web页面查看,其状态为StandBy

10.经典的wordcount
[root@louisvv01 ~]# hadoop jar
\ /opt/hadoop-2.7.3/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar
\ wordcount /test.txt /out1
17/12/06 15:38:06 INFO mapreduce.Job: map 0% reduce 0%
17/12/06 15:38:24 INFO mapreduce.Job: map 100% reduce 0%
17/12/06 15:38:56 INFO mapreduce.Job: map 100% reduce 100%
[root@louisvv01 ~]# hadoop fs -ls /out1/
Found 2 items
-rw-r–r– 3 root supergroup 0 2017-12-06 15:38 /out1/_SUCCESS
-rw-r–r– 3 root supergroup 37 2017-12-06 15:38 /out1/part-r-00000
[root@louisvv01 ~]# hadoop fs -cat /out1/part-r-00000
hello 4
louisvv 1
me 1
world 1
you 1
好了,本篇hadoop高可用集群搭建写完了
记得点赞和分享哦!

赫墨拉

我是一个喜爱大数据的小菜鸡,这里是我分享我的成长和经历的博客

发表评论

电子邮件地址不会被公开。