hadoop学习笔记（二）：centos7三节点安装hadoop2.7.0

环境win7+vamvare10+centos7一、新建三台centos764位的虚拟机master192.168.137.100root123456node1192.168.13

环境win7+vamvare10+centos7

一、新建三台centos7 64位的虚拟机

master 192.168.137.100 root/123456node1 192.168.137.101 root/123456node2 192.168.137.102 root/123456

 

二、关闭三台虚拟机的防火墙，在每台虚拟机里面执行：

systemctl stop firewalld.servicesystemctl disable firewalld.service

 

三、在三台虚拟机里面的/etc/hosts添加三行

192.168.137.100 master192.168.137.101 node1192.168.137.102 node2

 

四、为三台机器设置ssh免密登录

  1、CentOS7默认没有启动ssh无密登录，去掉/etc/ssh/sshd_config其中1行的注释，每台服务器都要设置

#PubkeyAuthentication yes

 然后重启ssh服务

systemctl restart sshd

 

2、在master机器的/root执行：ssh-keygen -t rsa命令，一直按回车。三台机器都要执行。

[root@master ~]# ssh-keygen -t rsaGenerating public/private rsa key pair.Enter file in which to save the key (/root/.ssh/id_rsa): Created directory '/root/.ssh'.Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /root/.ssh/id_rsa.Your public key has been saved in /root/.ssh/id_rsa.pub.The key fingerprint is:SHA256:aMUO8b/EkylqTMb9+71ePnQv0CWQohsaMeAbMH+t87M root@masterThe key's randomart image is:+---[RSA 2048]----+| o ... . || = o= . o || + oo=. . . || =.Boo o . .|| . OoSoB . o || =.+.+ o. ...|| + o o .. +|| . o . ..+.|| E ....+oo|+----[SHA256]-----+

  3、在master上合并公钥到authorized_keys文件

[root@master ~]# cd /root/.ssh/[root@master .ssh]# lltotal 8-rw-------. 1 root root 1679 Apr 19 11:10 id_rsa-rw-r--r--. 1 root root 393 Apr 19 11:10 id_rsa.pub[root@master .ssh]# cat id_rsa.pub>> authorized_keys

  4、将master的authorized_keys复制到node1和node2节点

scp /root/.ssh/authorized_keys root@192.168.137.101:/root/.ssh/scp /root/.ssh/authorized_keys root@192.168.137.102:/root/.ssh/

  5、测试：

[root@master ~]# ssh root@192.168.137.101Last login: Thu Apr 19 11:41:23 2018 from 192.168.137.100[root@node1 ~]#

 

[root@master ~]# ssh root@192.168.137.102Last login: Mon Apr 23 10:40:38 2018 from 192.168.137.1[root@node2 ~]#

 

五、为三台机器安装jdk

1、jdk下载地址：https://pan.baidu.com/s/1-fhy_zbGbEXR1SBK8V7aNQ

2、创建目录：/home/java

mkdir -p /home/java

3、将下载的文件jdk-7u79-linux-x64.tar.gz，放到/home/java底下，并执行以下命令：

tar -zxf jdk-7u79-linux-x64.tar.gzrm -rf tar -zxf jdk-7u79-linux-x64.tar.gz

4、配置环境变量：

 vi /etc/profile,添加以下内容

export JAVA_HOME=/home/java/jdk1.7.0_79export CLASSPATH=.:$JAVA_HOME/jre/lib/rt.jar:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jarexport PATH=$PATH:$JAVA_HOME/bin

然后:source /etc/profile

测试：

[root@master jdk1.7.0_79]# java -versionjava version "1.7.0_79"Java(TM) SE Runtime Environment (build 1.7.0_79-b15)Java HotSpot(TM) 64-Bit Server VM (build 24.79-b02, mixed mode)[root@master jdk1.7.0_79]#

 

六、安装hadoop 2.7（只在Master服务器解压，再复制到Slave服务器）

 1、创建/home/hadoop目录

mkdir -p /home/hadoop

2、将hadoop-2.7.0.tar.gz放到/home/hadoop下并解压

tar -zxf hadoop-2.7.0.tar.gz

3、在/home/hadoop目录下创建数据存放的文件夹，tmp、hdfs/data、hdfs/name

[root@master hadoop]# mkdir tmp[root@master hadoop]# mkdir -p hdfs/data[root@master hadoop]# mkdir -p hdfs/name

4、配置配置/home/hadoop/hadoop-2.7.0/etc/hadoop目录下的core-site.xml

fs.defaultFS hdfs://192.168.137.100:9000 fs.default.name hdfs://192.168.137.100:9000 hadoop.tmp.dir file:/home/hadoop/tmp io.file.buffer.size 131702

 

5、配置/home/hadoop/hadoop-2.7.0/etc/hadoop目录下的hdfs-site.xml

dfs.namenode.secondary.http-address 192.168.137.100:9001 dfs.namenode.name.dir file:/home/hadoop/dfs/name dfs.datanode.data.dir file:/home/hadoop/dfs/data dfs.replication 2 dfs.nameservices mycluster dfs.webhdfs.enabled true

6、将/home/hadoop/hadoop-2.7.0/etc/hadoop目录下的mapred-site.xml.template复制一份，并命名成mapred-site.xml

mapreduce.framework.name yarn mapreduce.jobhistory.address 192.168.137.100:10020 mapreduce.jobhistory.webapp.address 192.168.137.100:19888

7、配置/home/hadoop/hadoop-2.7.0/etc/hadoop目录下的yarn-site.xml

yarn.nodemanager.aux-services mapreduce_shuffle yarn.nodemanager.auxservices.mapreduce.shuffle.class org.apache.hadoop.mapred.ShuffleHandler yarn.resourcemanager.address 192.168.137.100:8032 yarn.resourcemanager.scheduler.address 192.168.137.100:8030 yarn.resourcemanager.resource-tracker.address 192.168.137.100:8031 yarn.resourcemanager.admin.address 192.168.137.100:8033 yarn.resourcemanager.webapp.address 192.168.137.100:8088 yarn.nodemanager.resource.memory-mb 768

8、配置/home/hadoop/hadoop-2.7.0/etc/hadoop目录下hadoop-env.sh、yarn-env.sh的JAVA_HOME，不设置的话，启动不了

export JAVA_HOME=/home/java/jdk1.7.0_79

9、配置/home/hadoop/hadoop-2.7.0/etc/hadoop目录下的slaves，删除默认的localhost，增加2个从节点

192.168.137.101192.168.137.102

10、将配置好的Hadoop复制到各个节点对应位置上

scp -r /home/hadoop 192.168.137.101:/home/scp -r /home/hadoop 192.168.137.102:/home/

11、在Master服务器启动hadoop，从节点会自动启动，进入/home/hadoop/hadoop-2.7.0

  1）初始化：bin/hdfs namenode -format

  2）全部启动sbin/start-all.sh，也可以分开sbin/start-dfs.sh、sbin/start-yarn.sh

 

[root@master hadoop-2.7.0]# sbin/start-all.shThis script is Deprecated. Instead use start-dfs.sh and start-yarn.shStarting namenodes on [master]master: starting namenode, logging to /home/hadoop/hadoop-2.7.0/logs/hadoop-root-namenode-master.out192.168.137.101: starting datanode, logging to /home/hadoop/hadoop-2.7.0/logs/hadoop-root-datanode-node1.out192.168.137.102: starting datanode, logging to /home/hadoop/hadoop-2.7.0/logs/hadoop-root-datanode-node2.outstarting yarn daemonsstarting resourcemanager, logging to /home/hadoop/hadoop-2.7.0/logs/yarn-root-resourcemanager-master.out192.168.137.101: starting nodemanager, logging to /home/hadoop/hadoop-2.7.0/logs/yarn-root-nodemanager-node1.out192.168.137.102: starting nodemanager, logging to /home/hadoop/hadoop-2.7.0/logs/yarn-root-nodemanager-node2.out

 3）停止的话，输入命令，sbin/stop-all.sh

 4）输入命令，jps，可以看到相关信息

[root@master hadoop-2.7.0]# jps1765 ResourceManager2025 Jps

12、浏览器查看

  1）resourcemanager.webapp.address的界面

 

 

 2）namenode的界面

 

【感谢龙石为本站提供信息共享平台 http://www.longshidata.com/pages/exchange.html】