Ambari-大数据平台的搭建利器-在Centos上离线部署Ambari
上一片文章整理了：如何在Docker上部署Ambari，没有详细的介绍如何离线部署Ambari
这篇文章，来介绍一下Ambari以及Ambari在Centos上的离线部署
首先，先来了解一下Ambari
Ambari 跟 Hadoop 等开源软件一样，也是 Apache Software Foundation 中的一个项目，并且是顶级项目。
就 Ambari 的作用来说，就是创建、管理、监视 Hadoop 的集群，这里的 Hadoop 指的是Hadoop 整个生态圈（例如 Hive，Hbase，Sqoop，Zookeeper 等）。
Ambari 就是为了让 Hadoop 以及相关的大数据软件更容易使用的一个工具。
Ambari 现在所支持的平台组件也越来越多，例如流行的 Spark，Storm 等计算框架，以及资源调度平台 YARN 等，都能轻松地通过 Ambari 来进行部署。
对于初学者来说，搭建一个Hadoop集群不是一个简单的事情，辛辛苦苦的安装，调试，这个组件起不来，那个报错，很麻烦。
在实际应用场景，传统的部署方式较繁琐，需要手动配置的东西多，这时大家就希望有一种方便快速的方式去部署大数据平台， 正好，Ambari满足了我们的需求。
Ambari的离线部署
可先在测试环境下进行模拟部署，然后在实际场景中应用。
先准备一台虚拟机，安装Centos6/7系统
1.设置IP地址，修改主机名
vim /etc/sysconfig/network HOSTNAME= node1
2.关闭防火墙，禁止防火墙开机自启
Centos 6:
chkconfig iptables off service iptables stop
Centos 7:
systemctl disable firewalld.service systemctl stop firewalld.service
3.关闭SELinux
[root@node1 ~]# vim /etc/sysconfig/selinux 将SELINUX=enforcing改为SELINUX=disabled，执行该命令后重启机器生效
4.配置Host文件
vim /etc/hosts 192.168.1.201 node1 192.168.1.202 node2 192.168.1.203 node3
5.SSH配置
[root@node1 ~]# ssh-keygen -t rsa Generating public/private rsa key pair. Enter file in which to save the key (/root/.ssh/id_rsa): Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /root/.ssh/id_rsa. Your public key has been saved in /root/.ssh/id_rsa.pub. The key fingerprint is: 4d:91:79:95:00:10:7b:c7:aa:c9:6e:cd:92:62:7f:e0 root@node1 The key's randomart image is: +--[ RSA 2048]----+ | ooo+.o.. | | .oo. . | | . o.o | | + o | | S o | | ..o | | .+= | | o.E + | | . +oo | +-----------------+
6.安装JDK1.8
export JAVA_HOME=/opt/jdk1.7.0_79 export PATH=$PATH:$JAVA_HOME/bin [root@node1 cloud]# java -version java version "1.8.0_91" Java(TM) SE Runtime Environment (build 1.8.0_91-b14) Java HotSpot(TM) 64-Bit Server VM (build 25.91-b14, mixed mode)
7.安装NTP
Centos 6:
yum install -y ntp
Complete!
[root@node1 cloud]# chkconfig --list ntpd
ntpd 0:off 1:off 2:off 3:off 4:off 5:off 6:off
[root@node1 cloud]# chkconfig ntpd on
[root@node1 cloud]# chkconfig --list ntpd
ntpd 0:off 1:off 2:on 3:on 4:on 5:on 6:off
[root@node1 cloud]# service ntpd start
Starting ntpd: [ OK ]
Centos 7:
yum install -y ntp chkconfig --list ntpd systemctl is-enabled ntpd systemctl enable ntpd systemctl start ntpd
8.关闭Linux的THP服务
如果不关闭transparent_hugepage，HDFS会因为这个性能严重受影响。 关闭transparent_hugepage方法是： vi /etc/grub.conf 添加 transparent_hugepage=never vi /etc/rc.local 添加： if test -f /sys/kernel/mm/transparent_hugepage/defrag; then echo never > /sys/kernel/mm/transparent_hugepage/defrag fi if test -f /sys/kernel/mm/transparent_hugepage/enabled; then echo never > /sys/kernel/mm/transparent_hugepage/enabled fi exit 0 重启之后，用下面的命令检查： cat /sys/kernel/mm/redhat_transparent_hugepage/enabled 有 [never]则表示THP被禁用 [root@node1 ~]# cat /sys/kernel/mm/redhat_transparent_hugepage/enabled always madvise [never]
9.禁止Linux离线更新
vim /etc/yum/pluginconf.d/refresh-packagekit.conf 添加：enabled=0
以上是每个节点的基本环境配置。
复制出两台同样环境的虚拟机，node2，node3
分别修改虚拟机主机名， hosts文件，在node1上将SSH秘钥拷贝到node2，node3节点，测试SSH登录。
10.安装Ambari
下面开始制作本地源来安装Ambari
制作本地源，只需要在主节点上进行安装
-
配置HTTP服务
配置HTTP服务：配置HTTP服务到系统层使其随系统自动启动 chkconfig htttpd on service httpd start
-
安装工具
安装工具：安装本地制作相关工具 yum install yum-utils createrepo yum-plugin-priorities –y vim /etc/yum/pluginconf.d/priorities.conf 添加gpgcheck=0
-
下载Amabri和HDP安装包到本地，下载地址：
CentOS 6： http://public-repo-1.hortonworks.com/ambari/centos6/2.x/updates/2.5.0.3/ambari-2.5.0.3-centos6.tar.gz http://public-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.6.0.3/HDP-2.6.0.3-centos6-rpm.tar.gz http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.21/repos/centos6/HDP-UTILS-1.1.0.21-centos6.tar.gz CentOS 7： http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.5.0.3/ambari-2.5.0.3-centos7.tar.gz http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.6.0.3/HDP-2.6.0.3-centos7-rpm.tar.gz http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.21/repos/centos7/HDP-UTILS-1.1.0.21-centos7.tar.gz
-
将三个下载好的包上传到主节点，解压到/var/www/html目录下：
tar -zxf ambari-2.5.0.3-centos6.tar.gz -C /var/www/html tar -zxf HDP-2.6.0.3-centos6-rpm.tar.gz -C /var/www/html tar -zxf ambari-2.5.0.3-centos6.tar.gz -C /var/www/html/
- 在/etc/yum.repos.d/目录下创建ambari.repo文件，文件内容如下：
#VERSION_NUMBER=2.5.0.3-7 [ambari-2.5.0.3] name=ambari Version - ambari-2.5.0.3 baseurl=http://node1/ambari/centos6/ gpgcheck=0 gpgkey=http://node1/ambari/centos6/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins enabled=1 priority=1
- 在/etc/yum.repos.d/目录下创建HDP.repo文件，文件内容如下：
[HDP-2.6.0.3-8] name=HDP-2.6.0.3-8 baseurl=http://node1/HDP/centos6/ gpgcheck=0 gpgkey=http://node1/HDP/centos6/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins enabled=1 priority=1
- 在/etc/yum.repos.d/目录下创建HDP-UTILS.repo文件，文件内容如下：
[HDP-UTILS-1.1.0.21] name=HDP-UTILS-1.1.0.21 baseurl=http://node1/HDP-UTILS-1.1.0.21-centos6/ gpgcheck=0 gpgkey=http://node1/HDP-UTILS-1.1.0.21-centos6/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins enabled=1 priority=1
-
yum 配置
yum clean all Loaded plugins: fastestmirror, security Cleaning repos: HDP-2.6.0.3-8 HDP-UTILS-1.1.0.21 ambari-2.5.0.3 Cleaning up Everything Cleaning up list of fastest mirrors yum makecache Loaded plugins: fastestmirror, security Determining fastest mirrors HDP-2.6.0.3-8 | 2.9 kB 00:00 HDP-2.6.0.3-8/filelists_db | 455 kB 00:00 HDP-2.6.0.3-8/primary_db | 77 kB 00:00 HDP-2.6.0.3-8/other_db | 15 kB 00:00 HDP-UTILS-1.1.0.21 | 2.9 kB 00:00 HDP-UTILS-1.1.0.21/filelists_db | 95 kB 00:00 HDP-UTILS-1.1.0.21/primary_db | 36 kB 00:00 HDP-UTILS-1.1.0.21/other_db | 18 kB 00:00 ambari-2.5.0.3 | 2.9 kB 00:00 ambari-2.5.0.3/filelists_db | 147 kB 00:00 ambari-2.5.0.3/primary_db | 8.5 kB 00:00 ambari-2.5.0.3/other_db | 1.3 kB 00:00 Metadata Cache Created yum list，显示ok，则本地源配置完毕
重点来了，安装Amabri-Server
-
安装Ambari-Server
yum install ambari-server #执行成功后，针对mysql数据库再做一些工作
- 安装mysql
[root@node1 ~]# yum install -y mysql-server，并设置mysql开机自启 Mysql-connector版本5.1.40
将mysql-connector-Java.jar复制到/usr/share/java目录下 mkdir /usr/share/java cp /opt/mysql-connector-java-5.1.40.jar /usr/share/java/mysql-connector-java.jar
将mysql-connector-java.jar复制到/var/lib/ambari-server/resources目录下 cp /usr/share/java/mysql-connector-java.jar /var/lib/ambari-server/resources/mysql-jdbc-driver.jar
编辑ambari.properties vi /etc/ambari-server/conf/ambari.properties 添加 server.jdbc.driver.path=/usr/share/java/mysql-connector-java.jar
-
配置mysql
#下面的语句，可自定义修改 CREATE DATABASE ambari; use ambari; CREATE USER 'ambari'@'%' IDENTIFIED BY 'ambari'; GRANT ALL PRIVILEGES ON *.* TO 'ambari'@'%'; CREATE USER 'ambari'@'localhost' IDENTIFIED BY 'ambari'; GRANT ALL PRIVILEGES ON *.* TO 'ambari'@'localhost'; CREATE USER 'ambari'@'node1' IDENTIFIED BY 'ambari'; GRANT ALL PRIVILEGES ON *.* TO 'ambari'@'node1'; FLUSH PRIVILEGES; source /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql show tables; use mysql; select Host,User,Password from user where user='ambari'; CREATE DATABASE hive; use hive; CREATE USER 'hive'@'%' IDENTIFIED BY 'hive'; GRANT ALL PRIVILEGES ON *.* TO 'hive'@'%'; CREATE USER 'hive'@'localhost' IDENTIFIED BY 'hive'; GRANT ALL PRIVILEGES ON *.* TO 'hive'@'localhost'; CREATE USER 'hive'@'node1' IDENTIFIED BY 'hive'; GRANT ALL PRIVILEGES ON *.* TO 'hive'@'node1'; FLUSH PRIVILEGES; CREATE DATABASE oozie; use oozie; CREATE USER 'oozie'@'%' IDENTIFIED BY 'oozie'; GRANT ALL PRIVILEGES ON *.* TO 'oozie'@'%'; CREATE USER 'oozie'@'localhost' IDENTIFIED BY 'oozie'; GRANT ALL PRIVILEGES ON *.* TO 'oozie'@'localhost'; CREATE USER 'oozie'@'node1' IDENTIFIED BY 'oozie'; GRANT ALL PRIVILEGES ON *.* TO 'oozie'@'node1'; FLUSH PRIVILEGES;
- 配置Ambari
执行ambari-server setup 下面是配置执行流程，按照提示操作 （1） 提示是否自定义设置。输入：y Customize user account for ambari-server daemon [y/n] (n)? y （2）ambari-server 账号。 Enter user account for ambari-server daemon (root): 如果直接回车就是默认选择root用户 如果输入已经创建的用户就会显示： Enter user account for ambari-server daemon (root):ambari Adjusting ambari-server permissions and ownership... （3）检查防火墙是否关闭 Adjusting ambari-server permissions and ownership... Checking firewall... WARNING: iptables is running. Confirm the necessary Ambari ports are accessible. Refer to the Ambari documentation for more details on ports. OK to continue [y/n] (y)? 直接回车 （4）设置JDK。输入：3 Checking JDK... Do you want to change Oracle JDK [y/n] (n)? y [1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8 [2] Oracle JDK 1.7 + Java Cryptography Extension (JCE) Policy Files 7 [3] Custom JDK ============================================================================== Enter choice (1): 3 如果上面选择3自定义JDK,则需要设置JAVA_HOME。输入：本地java安装目录地址 WARNING: JDK must be installed on all hosts and JAVA_HOME must be valid on all hosts. WARNING: JCE Policy files are required for configuring Kerberos security. If you plan to use Kerberos,please make sure JCE Unlimited Strength Jurisdiction Policy Files are valid on all hosts. Path to JAVA_HOME: /usr/java/jdk1.8.0_131 Validating JDK on Ambari Server...done. Completing setup... （5）数据库配置。选择：y Configuring database... Enter advanced database configuration [y/n] (n)? y （6）选择数据库类型。输入：3 Configuring database... ============================================================================== Choose one of the following options: [1] - PostgreSQL (Embedded) [2] - Oracle [3] - MySQL [4] - PostgreSQL [5] - Microsoft SQL Server (Tech Preview) [6] - SQL Anywhere ============================================================================== Enter choice (3): 3 （7）设置数据库的具体配置信息，根据实际情况输入，如果和括号内相同，则可以直接回车。如果想重命名，就输入。 Hostname (localhost):node1 Port (3306): Database name (ambari): Username (ambari): Enter Database Password (bigdata): Re-Enter password: （8）将Ambari数据库脚本导入到数据库 WARNING: Before starting Ambari Server, you must run the following DDL against the database to create the schema: /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql Proceed with configuring remote database connection properties [y/n] (y)? 如果使用自己定义的数据库，必须在启动Ambari服务之前导入Ambari的sql脚本。 执行语句： ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java.jar
-
启动Ambari
ambari-server start #启动ambari，Start成功后，在浏览器输入Ambari地址：http://node1:8080/
- 安装部署HDP集群
1.首先进行登录，用户名密码都是admin
2.登陆后，进入安装向导
点击Launch Install Wizard
3.选择版本，选择本地安装，输入本地安装的目录
4.配置集群节点，SSH配置，以及节点确认，agent安装
5.选择安装服务，请根据情况，自定义安装
6.服务节点分配
7.客户端自定义配置
这一步是对已选服务的自定义配置，需要设置密码的服务非常多。
这里要注意一下hive和oozie，如果已经在MySQL中创建了元表，选择已经存在的MySQL，输入URL和密码后，点击连接测试总是失败。需要先停止Ambari，执行下面这个命令：
ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java.jar
再重启Ambari后点击连接测试就会成功。
8.显示配置信息
所有信息配置完成后，会出现一份报告，汇总这次安装的集群节点规划情况
9.安装、启动和测试
这一步安装各个服务，并且完成安装后会启动和测试相关服务。这一步的时间比较长，如果中途出现错误，可根据具体提示或者log进行操作。
10.安装成功
如果都出现绿色的success便说明集群正确的安装成功。进入http://node1:8080/#/main/dashboard/metrics看到平台主页面了
谢谢博主，对我帮助很大，继续关注
写的很详细，学习了
I see you don’t monetize your blog, don’t waste your traffic,
you can earn additional cash every month because you’ve got high quality content.
If you want to know how to make extra $$$, search for: Boorfe’s tips best adsense alternative
No, thank you
I have checked your site and i’ve found some duplicate content, that’s why
you don’t rank high in google, but there is a tool that
can help you to create 100% unique articles, search
for; Boorfe’s tips unlimited content
no,thanks
==========================
Creating target directory…
==========================
Command start time 2018-12-29 16:25:40
Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).
SSH command execution finished
host=192.168.147.121, exitcode=255
Command end time 2018-12-29 16:25:40
ERROR: Bootstrap of host 192.168.147.121 fails because previous action finished with non-zero exit code (255)
ERROR MESSAGE: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).
STDOUT:
Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).
这个问题，需要检查一下相关的用户权限。