当前位置 主页 > 服务器问题 > Linux/apache问题 >

    centos7搭建hadoop2.10高可用(HA)

    栏目:Linux/apache问题 时间:2020-02-07 11:41

    本篇介绍在centos7中搭建hadoop2.10高可用集群,首先准备6台机器:2台nn(namenode);4台dn(datanode);3台jns(journalnodes)

    IP hostname 进程
    192.168.30.141 s141 nn1(namenode),zkfc(DFSZKFailoverController),zk(QuorumPeerMain)
    192.168.30.142 s142 dn(datanode), jn(journalnode),zk(QuorumPeerMain)
    192.168.30.143 s143 dn(datanode), jn(journalnode),zk(QuorumPeerMain)
    192.168.30.144 s144 dn(datanode), jn(journalnode)
    192.168.30.145 s145 dn(datanode)
    192.168.30.146 s146 nn2(namenode),zkfc(DFSZKFailoverController)

    各个机器 jps进程:

    由于本人使用的是vmware虚拟机,所以在配置好一台机器后,使用克隆,克隆出剩余机器,并修改hostname和IP,这样每台机器配置就都统一了每台机器配置添加hdfs用户及用户组,配置jdk环境,安装hadoop,本次搭建高可用集群在hdfs用户下,可以参照:centos7搭建hadoop2.10伪分布模式

    下面是安装高可用集群的一些步骤和细节:

    1.设置每台机器的hostname 和 hosts

    修改hosts文件,hosts设置有后可以使用hostname访问机器,这样比较方便,修改如下:

    127.0.0.1 locahost
    192.168.30.141 s141
    192.168.30.142 s142
    192.168.30.143 s143
    192.168.30.144 s144
    192.168.30.145 s145
    192.168.30.146 s146

    2.设置ssh无密登录,由于s141和s146都为namenode,所以要将这两台机器无密登录到所有机器,最好hdfs用户和root用户都设置无密登录

    我们将s141设置为nn1,s146设置为nn2,就需要s141、s146能够通过ssh无密登录到其他机器,这样就需要在s141和s146机器hdfs用户下生成密钥对,并将s141和s146公钥发送到其他机器放到~/.ssh/authorized_keys文件中,更确切的说要将公钥添加的所有机器上(包括自己)

    在s141和s146机器上生成密钥对:

    ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa

    将id_rsa.pub文件内容追加到s141-s146机器的/home/hdfs/.ssh/authorized_keys中,现在其他机器暂时没有authorized_keys文件,我们就将id_rsa.pub更名为authorized_keys即可,如果其他机器已存在authorized_keys文件可以将id_rsa.pub内容追加到该文件后,远程复制可以使用scp命令:

    s141机器公钥复制到其他机器

    scp id_rsa.pub hdfs@s141:/home/hdfs/.ssh/id_rsa_141.pub
    scp id_rsa.pub hdfs@s142:/home/hdfs/.ssh/id_rsa_141.pub
    scp id_rsa.pub hdfs@s143:/home/hdfs/.ssh/id_rsa_141.pub
    scp id_rsa.pub hdfs@s144:/home/hdfs/.ssh/id_rsa_141.pub
    scp id_rsa.pub hdfs@s145:/home/hdfs/.ssh/id_rsa_141.pub
    scp id_rsa.pub hdfs@s146:/home/hdfs/.ssh/id_rsa_141.pub

    s146机器公钥复制到其他机器

    scp id_rsa.pub hdfs@s141:/home/hdfs/.ssh/id_rsa_146.pub
    scp id_rsa.pub hdfs@s142:/home/hdfs/.ssh/id_rsa_146.pub
    scp id_rsa.pub hdfs@s143:/home/hdfs/.ssh/id_rsa_146.pub
    scp id_rsa.pub hdfs@s144:/home/hdfs/.ssh/id_rsa_146.pub
    scp id_rsa.pub hdfs@s145:/home/hdfs/.ssh/id_rsa_146.pub
    scp id_rsa.pub hdfs@s146:/home/hdfs/.ssh/id_rsa_146.pub