在Docker上部署Ambari
最近在搞Docker上部署Ambari
此文梳理了一下整个部署的过程以及部署中遇到的问题和解决办法
下一篇博文,会写:如何在Centos上离线部署Ambari 2.5.0.3
登录服务器,查看服务器基本信息
[root@ai-main ~]# df -Th Filesystem Type Size Used Avail Use% Mounted on /dev/sda2 xfs 100G 40G 61G 40% / devtmpfs devtmpfs 16G 0 16G 0% /dev tmpfs tmpfs 16G 20K 16G 1% /dev/shm tmpfs tmpfs 16G 106M 16G 1% /run tmpfs tmpfs 16G 0 16G 0% /sys/fs/cgroup /dev/sda5 xfs 815G 3.6G 811G 1% /data /dev/sda1 xfs 497M 143M 354M 29% /boot tmpfs tmpfs 3.2G 0 3.2G 0% /run/user/0 [root@ai-main ~]# uname -r 3.10.0-514.el7.x86_64 [root@ai-main ~]# more /etc/redhat-release CentOS Linux release 7.3.1611 (Core)
关闭防火墙,关闭SELinux
关闭THP服务,使用两个命令查看
[root@ai-main selinux]# cat /sys/kernel/mm/transparent_hugepage/defrag [always] madvise never [root@ai-main selinux]# cat /sys/kernel/mm/transparent_hugepage/enabled [always] madvise never 这个状态就是启用的
编辑rc.local文件
vim /etc/rc.d/rc.local 增加下列内容: if test -f /sys/kernel/mm/transparent_hugepage/enabled; then echo never > /sys/kernel/mm/transparent_hugepage/enabled fi if test -f /sys/kernel/mm/transparent_hugepage/defrag; then echo never > /sys/kernel/mm/transparent_hugepage/defrag fi 保存退出,然后赋予rc.local文件执行权限: chmod +x /etc/rc.d/rc.local 最后重启系统,以后再检查THP,下面这样就是被禁用了 [root@ai-main ~]# cat /sys/kernel/mm/transparent_hugepage/enabled always madvise [never] [root@ai-main ~]# cat /sys/kernel/mm/transparent_hugepage/defrag always madvise [never]
生成SSH秘钥
[root@ai-main ~]# ssh-keygen -t rsa
安装http
安装httpd [root@ai-main ~]# yum install httpd -y Installed: httpd.x86_64 0:2.4.6-45.el7.centos.4 Dependency Installed: apr.x86_64 0:1.4.8-3.el7 apr-util.x86_64 0:1.5.2-6.el7 httpd-tools.x86_64 0:2.4.6-45.el7.centos.4 mailcap.noarch 0:2.1.41-2.el7 Complete!
在/etc/yum.repo.d/下创建ambari.repo文件
ambari.repo内容如下: [HDP-2.6.0.3-8] name=HDP-2.6.0.3-8 baseurl=http://ai-main/HDP/centos7/ gpgcheck=0 gpgkey=http://ai-main/HDP/centos7/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins enabled=1 priority=1
在/etc/yum.repo.d/下创建HDP.repo文件
HDP.repo文件内容如下 [HDP-2.6.0.3-8] name=HDP-2.6.0.3-8 baseurl=http://ai-main/HDP/centos7/ gpgcheck=0 gpgkey=http://ai-main/HDP/centos7/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins enabled=1 priority=1
在/etc/yum.repo.d/下创建HDP-UTILS.repo文件
HDP-UTILS.repo文件内容如下: [HDP-UTILS-1.1.0.21] name=HDP-UTILS-1.1.0.21 baseurl=http://ai-main/HDP-UTILS-1.1.0.21-centos7/ gpgcheck=0 gpgkey=http://ai-main/HDP-UTILS-1.1.0.21-centos7/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins enabled=1 priority=1
开始安装ambari
[root@ai-main yum.repos.d]# yum install -y ambari-server Installed: ambari-server.x86_64 0:2.5.0.3-7 Complete!
安装mysql
#yum install mysql-server 安装mysql-server失败,如下: [root@ai-main java]# yum install mysql-server Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile * base: mirrors.sina.cn * extras: mirrors.sina.cn * updates: mirrors.sina.cn No package mysql-server available. Error: Nothing to do
有两种解决办法:
1.安装MariaDB数据库
这里提一嘴,MariaDB数据库
MariaDB数据库管理系统是MySQL的一个分支,主要由开源社区在维护,采用GPL授权许可。
开发这个分支的原因之一是:甲骨文公司收购了MySQL后,有将MySQL闭源的潜在风险,因此社区采用分支的方式来避开这个风险。
MariaDB的目的是完全兼容MySQL,包括API和命令行,使之能轻松成为MySQL的代替品。
安装mariadb后显示的也是 MariaDB [(none)]> ,看起来有点不习惯。于是使用了下面的办法。
2.官网安装mysql
[root@ai-main java]# wget http://dev.mysql.com/get/mysql-community-release-el7-5.noarch.rpm 安装mysql有两种方式,一种是安装mariadb,另一种使用网络安装员 [root@ai-main java]# rpm -ivh mysql-community-release-el7-5.noarch.rpm Installed: mysql-community-client.x86_64 0:5.6.37-2.el7 mysql-community-libs.x86_64 0:5.6.37-2.el7 mysql-community-server.x86_64 0:5.6.37-2.el7
安装成功后,重启服务,登录mysql
[root@ai-main java]# service mysqld restart
[root@ai-main ~]# chkconfig mysqld on #设置mysql开机自启
[root@ai-main java]# mysql -u root -p #登录mysql Enter password: 默认没有密码,回车即可 Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 2 Server version: 5.6.37 MySQL Community Server (GPL) mysql> show databases; +--------------------+ | Database | +--------------------+ | information_schema | | mysql | | performance_schema | +--------------------+ 3 rows in set (0.00 sec) mysql>
在mysql中使用下面的语句
CREATE DATABASE ambari; use ambari; CREATE USER 'ambari'@'%' IDENTIFIED BY 'ambari'; GRANT ALL PRIVILEGES ON *.* TO 'ambari'@'%'; CREATE USER 'ambari'@'localhost' IDENTIFIED BY 'ambari'; GRANT ALL PRIVILEGES ON *.* TO 'ambari'@'localhost'; CREATE USER 'ambari'@'ai-main' IDENTIFIED BY 'ambari'; GRANT ALL PRIVILEGES ON *.* TO 'ambari'@'ai-main'; FLUSH PRIVILEGES; source /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql show tables; use mysql; select Host,User,Password from user where user='ambari'; CREATE DATABASE hive; use hive; CREATE USER 'hive'@'%' IDENTIFIED BY 'hive'; GRANT ALL PRIVILEGES ON *.* TO 'hive'@'%'; CREATE USER 'hive'@'localhost' IDENTIFIED BY 'hive'; GRANT ALL PRIVILEGES ON *.* TO 'hive'@'localhost'; CREATE USER 'hive'@'ai-main' IDENTIFIED BY 'hive'; GRANT ALL PRIVILEGES ON *.* TO 'hive'@'ai-main'; FLUSH PRIVILEGES; CREATE DATABASE oozie; use oozie; CREATE USER 'oozie'@'%' IDENTIFIED BY 'oozie'; GRANT ALL PRIVILEGES ON *.* TO 'oozie'@'%'; CREATE USER 'oozie'@'localhost' IDENTIFIED BY 'oozie'; GRANT ALL PRIVILEGES ON *.* TO 'oozie'@'localhost'; CREATE USER 'oozie'@'ai-main' IDENTIFIED BY 'oozie'; GRANT ALL PRIVILEGES ON *.* TO 'oozie'@'ai-main'; FLUSH PRIVILEGES;
安装ambari-server
[root@ai-main opt]# ambari-server setup #开始安装ambari-server Using python /usr/bin/python Setup ambari-server Checking SELinux... SELinux status is 'disabled' Customize user account for ambari-server daemon [y/n] (n)? y # Enter user account for ambari-server daemon (root): Adjusting ambari-server permissions and ownership... Checking firewall status... Checking JDK... [1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8 [2] Oracle JDK 1.7 + Java Cryptography Extension (JCE) Policy Files 7 [3] Custom JDK ============================================================================== Enter choice (1): 3 WARNING: JDK must be installed on all hosts and JAVA_HOME must be valid on all hosts. WARNING: JCE Policy files are required for configuring Kerberos security. If you plan to use Kerberos,please make sure JCE Unlimited Strength Jurisdiction Policy Files are valid on all hosts. Path to JAVA_HOME: /opt/jdk1.8.0_91 Validating JDK on Ambari Server...done. Completing setup... Configuring database... Enter advanced database configuration [y/n] (n)? y Configuring database... ============================================================================== Choose one of the following options: [1] - PostgreSQL (Embedded) [2] - Oracle [3] - MySQL / MariaDB [4] - PostgreSQL [5] - Microsoft SQL Server (Tech Preview) [6] - SQL Anywhere [7] - BDB ============================================================================== Enter choice (1): 3 Hostname (localhost): ai-main Port (3306): Database name (ambari): Username (ambari): Enter Database Password (bigdata): Re-enter password: Configuring ambari database... Copying JDBC drivers to server resources... Configuring remote database connection properties... WARNING: Before starting Ambari Server, you must run the following DDL against the database to create the schema: /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql Proceed with configuring remote database connection properties [y/n] (y)? y Extracting system views... ambari-admin-2.5.0.3.7.jar ........... Adjusting ambari-server permissions and ownership... Ambari Server 'setup' completed successfully. [root@ai-main opt]# clear [root@ai-main opt]# ambari-server start Using python /usr/bin/python Starting ambari-server Ambari Server running with administrator privileges. Organizing resource files at /var/lib/ambari-server/resources... Ambari database consistency check started... Server PID at: /var/run/ambari-server/ambari-server.pid Server out at: /var/log/ambari-server/ambari-server.out Server log at: /var/log/ambari-server/ambari-server.log Waiting for server start.................... Server started listening on 8080 DB configs consistency check: no errors and warnings were found. Ambari Server 'start' completed successfully.
到这里在宿主机Ambari环境准备完成了
ambari搭建的主机如下:
ai-main 主机
ai-node1 docker容器1,从节点node1
ai-node2 docker容器2,从节点node2
下面开始安装docker,配置docker环境
机器分布如下:
安装docker
安装docker [root@ai-main ~]# yum install docker-io -y Installed: docker.x86_64 2:1.12.6-32.git88a4867.el7.centos Complete!
启动docker
[root@ai-main ~]# service docker start
[root@ai-main opt]# docker search centos #搜索centos系统 INDEX NAME DESCRIPTION STARS OFFICIAL AUTOMATED docker.io docker.io/centos The official build of CentOS. 3608 [OK] docker.io docker.io/ansible/centos7-ansible Ansible on Centos7 100 [OK] docker.io docker.io/jdeathe/centos-ssh CentOS-6 6.9 x86_64 / CentOS-7 7.3.1611 x8... 81 [OK] docker.io docker.io/tutum/centos Simple CentOS docker image with SSH access 33 docker.io docker.io/imagine10255/centos6-lnmp-php56 centos6-lnmp-php56 30 [OK] docker.io docker.io/gluster/gluster-centos Official GlusterFS Image [ CentOS-7 + Glu... 20 [OK] docker.io docker.io/kinogmt/centos-ssh CentOS with SSH 16 [OK] docker.io docker.io/centos/php-56-centos7 PHP 5.6 platform for building and running ... 9 docker.io docker.io/guyton/centos6 From official centos6 container with full ... 7 [OK] docker.io docker.io/openshift/base-centos7 A Centos7 derived base image for Source-To... 7 docker.io docker.io/openshift/mysql-55-centos7 DEPRECATED: A Centos7 based MySQL v5.5 ima... 6 docker.io docker.io/centos/python-35-centos7 Python 3.5 platform for building and runni... 5 docker.io docker.io/darksheer/centos Base Centos Image -- Updated hourly 3 [OK] docker.io docker.io/openshift/jenkins-2-centos7 A Centos7 based Jenkins v2.x image for use... 3 docker.io docker.io/openshift/ruby-20-centos7 DEPRECATED: A Centos7 based Ruby v2.0 imag... 3 docker.io docker.io/blacklabelops/centos CentOS Base Image! Built and Updates Daily! 1 [OK] docker.io docker.io/indigo/centos-maven Vanilla CentOS 7 with Oracle Java Developm... 1 [OK] docker.io docker.io/openshift/php-55-centos7 DEPRECATED: A Centos7 based PHP v5.5 image... 1 docker.io docker.io/pivotaldata/centos-gpdb-dev CentOS image for GPDB development. Tag nam... 1 docker.io docker.io/pivotaldata/centos-mingw Using the mingw toolchain to cross-compile... 1 docker.io docker.io/jameseckersall/sonarr-centos Sonarr on CentOS 7 0 [OK] docker.io docker.io/miko2u/centos6 CentOS6 日本語環境 0 [OK] docker.io docker.io/pivotaldata/centos Base centos, freshened up a little with a ... 0 docker.io docker.io/pivotaldata/centos-gcc-toolchain CentOS with a toolchain, but unaffiliated ... 0 docker.io docker.io/smartentry/centos centos with smartentry 0 [OK] [root@ai-main opt]# docker pull centos #pull centos的镜像 Using default tag: latest Trying to pull repository docker.io/library/centos ... latest: Pulling from docker.io/library/centos 74f0853ba93b: Pull complete Digest: sha256:26f74cefad82967f97f3eeeef88c1b6262f9b42bc96f2ad61d6f3fdf544759b8
在Ambari上部署,docker需要指定IP
要自定义docker网络,并需要指定一个空闲的网段
#创建自定义网络,并指定网段 [root@ai-main opt]# docker network create --subnet=172.18.0.0/24 ambari-network #这里,需要指定一个空闲的网段,原本以为可以使用和主机一样的网段,结果,网络瞬间挂了,还得去服务器那里修改回来,需要注意 17025d3fdfb1d30691cded810aaaafbbce9eb6813e9ef4d2bb499cc36dd94f9d [root@ai-main ~]# docker network ls NETWORK ID NAME DRIVER SCOPE 3f86861e1b75 ambari-network bridge local [root@ai-main ~]# ifconfig #可以看到自定义的docker网段 br-3f86861e1b75: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 172.18.0.1 netmask 255.255.255.0 broadcast 0.0.0.0 inet6 fe80::42:31ff:fe9b:78d0 prefixlen 64 scopeid 0x20<link> ether 02:42:31:9b:78:d0 txqueuelen 0 (Ethernet) RX packets 0 bytes 0 (0.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 0 bytes 0 (0.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
创建好网络后,就要进入docker容器进行环境安装了
docker run -it --net ambari-network --ip 172.18.0.10 centos:latest
这个centos是什么都没有的,都需要自己进行安装,进去查看ip地址,查看不了,需要安装net-tools
yum install net-tools -y
安装后,使用ifconfig就可以查看IP了
安装SSH
先修改root密码: [root@5f58f4e94fb8 sshd]# passwd root Changing password for user root. New password: BAD PASSWORD: The password is shorter than 8 characters Retype new password: passwd: all authentication tokens updated successfully.
[root@5f58f4e94fb8 opt]# yum install -y openssh-server #安装ssh Installed: openssh-server.x86_64 0:6.6.1p1-35.el7_3 Dependency Installed: fipscheck.x86_64 0:1.4.1-5.el7 fipscheck-lib.x86_64 0:1.4.1-5.el7 openssh.x86_64 0:6.6.1p1-35.el7_3 tcp_wrappers-libs.x86_64 0:7.6-77.el7 Complete!
在启动ssh中出现了问题
#安装好ssh后,需要在/var/run/下创建sshd目录 [root@5f58f4e94fb8 opt]# mkdir -p /var/run/sshd #创建好目录后,启动ssh [root@5f58f4e94fb8 sshd]# /usr/sbin/sshd -D & [1] 204 #却发现了报错 [root@5f58f4e94fb8 sshd]# Could not load host key: /etc/ssh/ssh_host_rsa_key Could not load host key: /etc/ssh/ssh_host_ecdsa_key Could not load host key: /etc/ssh/ssh_host_ed25519_key
解决办法:输入以下命令
root@5f58f4e94fb8 sshd]# ssh-keygen -q -t rsa -b 2048 -f /etc/ssh/ssh_host_rsa_key -N '' [root@5f58f4e94fb8 sshd]# ssh-keygen -q -t ecdsa -f /etc/ssh/ssh_host_ecdsa_key -N '' [root@5f58f4e94fb8 sshd]# ssh-keygen -t dsa -f /etc/ssh/ssh_host_ed25519_key -N '' Generating public/private dsa key pair. Your identification has been saved in /etc/ssh/ssh_host_ed25519_key. Your public key has been saved in /etc/ssh/ssh_host_ed25519_key.pub. The key fingerprint is: 32:aa:9c:11:22:86:1d:53:eb:55:e6:67:fa:d0:3e:2e root@5f58f4e94fb8 The key's randomart image is: +--[ DSA 1024]----+ | . o | | . . + | | o . . . o | |.. + . = | |+.o . o S . | |o. . . o + | | . . + | | . + E. . | | + .. | +-----------------+ [root@5f58f4e94fb8 sshd]# /usr/sbin/sshd -D & #ssh启动正常 [2] 208
启动ssh后,需要在主机上将ssh秘钥拷贝到docker节点
[root@ai-main opt]# ssh-copy-id 172.18.0.10 The authenticity of host '172.18.0.10 (172.18.0.10)' can't be established. ECDSA key fingerprint is 56:0b:cb:cd:33:85:6e:74:6c:02:50:91:4c:9c:c1:82. Are you sure you want to continue connecting (yes/no)? yes /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys root@172.18.0.10's password: Number of key(s) added: 1 Now try logging into the machine, with: "ssh '172.18.0.10'" and check to make sure that only the key(s) you wanted were added.
安装java环境
不需要使用网络源下载安装java,将本地的java包复制到docker上,解压,配置环境变量,并source
docker与host容器互传文件(数据) docker cp 文件路径 容器id:/容器目录 [root@ai-main opt]# docker cp /opt/jdk-8u91-linux-x64.tar.gz 5f58f4e94fb8:/opt/ #需要在主机上操作
基础环境准备好了,将容器commit为一个镜像ambari-env,然后启动两个容器
这里重点要说一下docker的初始化配置
1.需要指定主机名 -h 参数,要不然主机名自动生成
2.需要指定前面配置好的专属网络,指定IP地址 –net 参数指定网络 –ip参数 指定ip
3. -v /etc/localtime:/etc/localtime:ro 参数是设置容器时间与主机时间同步,会在文章后面说明
4.需要添加 -d 参数使其在后台运行
5.容器开启后,进入容器,查看ip地址,修改 /ets/hosts映射文件
6.需要后台开启ssh服务
7.退出时,不能使用ctrl c或者exit退出,这样容器就会停止,使用 ctrl + q+p退出容器
启动两个容器 docker run -itd --name ai-node1 --net ambari-network --ip 172.18.0.10 -h ai-node1 -v /etc/localtime:/etc/localtime:ro ambari-env 25f01601c95ffd6bb2d4f4f58f50281d1d7c3cb834e13c2f69a43cbee5737f13 [root@ai-main ~]# docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 25f01601c95f ambari-env "/bin/bash" 7 seconds ago Up 6 seconds ai-node1 3f18c6919166 ambari-env "/bin/bash" About a minute ago Up About a minute ai-node2 5f58f4e94fb8 docker.io/centos "/bin/bash" 19 hours ago Exited (0) 18 hours ago node1 1751cc6aa7d2 docker.io/centos "/bin/bash" 19 hours ago Exited (0) 19 hours ago elegant_hamilton [root@ai-main ~]# docker attach 25f01601c95f [root@ai-node1 /]# ifconfig eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 172.18.0.10 netmask 255.255.255.0 broadcast 0.0.0.0 inet6 fe80::42:acff:fe12:a prefixlen 64 scopeid 0x20<link> ether 02:42:ac:12:00:0a txqueuelen 0 (Ethernet) RX packets 8 bytes 648 (648.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 8 bytes 648 (648.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 [root@ai-node1 /]# vi /etc/hosts [root@ai-node1 /]# /usr/sbin/sshd -D & [1] 18
容器都开启后,测试与主机的连通性,ssh登录,测试完毕后,回到主机
Ambari安装
选择HDP版本,选择本地源安装,填写本地目录
点击next,上传ssh秘钥文件,下一步
连通性测试
在这个连通,安装的过程中,遇到了问题
ERROR 2017-09-05 17:13:46,069 main.py:244 - Ambari agent machine hostname (ai-main.insightcredit) does not match expected ambari server hostname (ai-main). Aborting registration. Please check hostname, hostname -f and /etc/hosts file to confirm your hostname is setup correctly 主机名和/etc/hosts中的主机名不对应,我去,平时还没注意 [root@ai-main HDP-UTILS-1.1.0.21-centos7]# hostname ai-main.insightcredit [root@ai-main HDP-UTILS-1.1.0.21-centos7]# hostname ai-main #临时修改主机名 vim /etc/sysconfig/network 修改主机名,重启即可 [root@ai-main HDP-UTILS-1.1.0.21-centos7]# hostname ai-main
下面就是集群角色分配,初始化配置,安装
在安装中遇到了如下问题
Error: Error: Unable to run the custom hook script ['/usr/bin/python', '/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-ANY/scripts/hook.py', 'ANY', '/var/lib/ambari-agent/data/command-279.json', '/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-ANY', '/var/lib/ambari-agent/data/structured-out-279.json', 'INFO', '/var/lib/ambari-agent/tmp', 'PROTOCOL_TLSv1'] 安装时间不一致,这机器时间明显就不一样。。。。
在上面docker初始化提到了,添加-v /etc/localtime:/etc/localtime:ro参数是为了和主机时间同步
在弄这里的时候,博主将docker容器关掉之后,重新启动的,很费事
最后,只需要等待安装就好了
安装后的页面,还有个别小问题需要再去搞一搞
总结一下:
此文介绍了在Docker上部署ambari,如果需要快速搭建开发环境可以参考,但是实际环境中,并不推荐使用,使用docker存数据不太靠谱,集群是伪分布式,服务器资源占用率高
注意:
在重启docker容器时,需要重新开启ssh,重新修改/etc/hosts文件,修改hosts文件后,启动ambari-agent服务,source /etc/profile文件。要不然在ambari中显示主机丢失心跳包,找不到该主机。
可以,有机会我也参考着撘一套。
看起来都高大上,哈哈
您好,有幸拜读了您的文章,目前在学习用三台服务器要搭建一个大数据实验环境用,有两个问题想请教您 1.用docker + ambari这种方式部署和直接服务器上部署,性能上差别大么?
2.您推荐的这种方法。可以直接部署在三台服务器上么? (每个服务器一个docer这种好呢还是 每个服务器有几个docker节点 然后把所有的docker节点建立统一的集群这种好?)三台服务器两cpu12核/64G内存/4T硬盘 方便留个QQ请教一下么?
您好,有幸拜读了您的文章,目前在学习用三台服务器要搭建一个大数据实验环境用,有两个问题想请教您 1.用docker + ambari这种方式部署和直接服务器上部署,性能上差别大么?
2.您推荐的这种方法。可以直接部署在三台服务器上么? (每个服务器一个docer这种好呢还是 每个服务器有几个docker节点 然后把所有的docker节点建立统一的集群这种好?)三台服务器两cpu12核/64G内存/4T硬盘 方便留个QQ请教一下么?
感谢您的留言。
当时我在写这篇文章的时候,服务器资源有限,仅有一台服务器,但想在该服务器上实现集群部署,于是使用了docker + ambari。
现在回答您的问题:
1.正如我当时的情况,使用docker+ambari的前提是,服务器资源有限,但想实现集群部署,可以使用这种方法。
关于性能问题:使用docker容器作为集群节点,将会共享宿主机的资源,性能会大打折扣
在实际应用环境中,不建议使用docker,涉及到系统稳定性和安全性,服务器远比docker可靠,但ambari可以作为搭建、管理大数据平台的一种选择。
2.鉴于您的服务器资源情况,不推荐使用docker + ambari的形式,直接使用ambari在服务器上搭建集群即可。
磁盘不要组raid,datanode挂多个硬盘,提高写速度,单独搭建好点,vmvar的 esxi能把你的物理机搭建成云平台,找个key 以上紧代表个人意见
磁盘不要组raid,datanode挂多个硬盘,提高写速度,单独搭建好点,vmvar的 esxi能把你的物理机搭建成云平台,找个key 以上紧代表个人意见
可以可以!