Hadoop伪分布式情况搭建

文昭关情

微信公众号：大佛拈花
关注可相识更多的教程及排版本领。题目或发起，请公众号留言;

1 硬件信息

1vCPUs | 1GB | s3.small.1
CentOS 7.2 64bit
2 Java情况

1 安装JDK8

1 下载安装

sudo yum install java-1.8.0-openjdk java-1.8.0-openjdk-devel
2 设置情况变量

vi ~/.bashrc
1export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk
2
3生存文件后实行下面下令使 JAVA_HOME 情况变量见效:
4source ~/.bashrc
测试：java -version
1[root@gosaint ~]# java -version
2openjdk version "1.8.0_191"
3OpenJDK Runtime Environment (build 1.8.0_191-b12)
4OpenJDK 64-Bit Server VM (build 25.191-b12, mixed mode)
3  安装hadoop

1 下载安装

1wget https://archive.apache.org/dist/hadoop/common/hadoop-2.7.4/hadoop-2.7.4.tar.gz
2 文件重定名

1cd /usr/local
2mv ./hadoop-2.7.4/ ./hadoop
3 检测hadoop是否安装

/usr/local/hadoop/bin/hadoop version
4 情况变量设置

1vi ~/.bashrc
2追加：
3export HADOOP_HOME=/usr/local/hadoop
4export HADOOP_INSTALL=$HADOOP_HOME
5export HADOOP_MAPRED_HOME=$HADOOP_HOME
6export HADOOP_COMMON_HOME=$HADOOP_HOME
7export HADOOP_HDFS_HOME=$HADOOP_HOME
8export YARN_HOME=$HADOOP_HOME
9export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
10export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin
11情况变量见效
12source ~/.bashrc
5 修改 Hadoop 的设置文件

Hadoop的设置文件位于安装目次的 /etc/hadoop 目次下，在本教程中即位于 /url/local/hadoop/etc/hadoop 目次下，必要修改的设置文件为如下两个:
1/usr/local/hadoop/etc/hadoop/core-site.xml
2/usr/local/hadoop/etc/hadoop/hdfs-site.xml
编辑 core-site.xml，修改节点的内容为如下所示：
core-site.xml
1<span class="hljs-tag" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">
2    <span class="hljs-tag" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">
3        <span class="hljs-tag" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">hadoop.tmp.dir</span class="hljs-name" style="font-size: inherit;line-height: inherit;color: rgb(248, 35, 117);overflow-wrap: inherit !important;word-break: inherit !important;"name/span>
4        <span class="hljs-tag" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">file:/usr/local/hadoop/tmp</span class="hljs-name" style="font-size: inherit;line-height: inherit;color: rgb(248, 35, 117);overflow-wrap: inherit !important;word-break: inherit !important;"value/span>
5        <span class="hljs-tag" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">location to store temporary files</span class="hljs-name" style="font-size: inherit;line-height: inherit;color: rgb(248, 35, 117);overflow-wrap: inherit !important;word-break: inherit !important;"description/span>
6    </span class="hljs-name" style="font-size: inherit;line-height: inherit;color: rgb(248, 35, 117);overflow-wrap: inherit !important;word-break: inherit !important;"property/span>
7    <span class="hljs-tag" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">
8        <span class="hljs-tag" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">fs.defaultFS</span class="hljs-name" style="font-size: inherit;line-height: inherit;color: rgb(248, 35, 117);overflow-wrap: inherit !important;word-break: inherit !important;"name/span>
9        <span class="hljs-tag" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">hdfs://10.112.38.11:9000</span class="hljs-name" style="font-size: inherit;line-height: inherit;color: rgb(248, 35, 117);overflow-wrap: inherit !important;word-break: inherit !important;"value/span>
10    </span class="hljs-name" style="font-size: inherit;line-height: inherit;color: rgb(248, 35, 117);overflow-wrap: inherit !important;word-break: inherit !important;"property/span>
11</span class="hljs-name" style="font-size: inherit;line-height: inherit;color: rgb(248, 35, 117);overflow-wrap: inherit !important;word-break: inherit !important;"configuration/span>
同理，编辑 hdfs-site.xml，修改节点的内容为如下所示：
1<span class="hljs-tag" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">
2    <span class="hljs-tag" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">
3        <span class="hljs-tag" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">dfs.replication</span class="hljs-name" style="font-size: inherit;line-height: inherit;color: rgb(248, 35, 117);overflow-wrap: inherit !important;word-break: inherit !important;"name/span>
4        <span class="hljs-tag" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">1</span class="hljs-name" style="font-size: inherit;line-height: inherit;color: rgb(248, 35, 117);overflow-wrap: inherit !important;word-break: inherit !important;"value/span>
5    </span class="hljs-name" style="font-size: inherit;line-height: inherit;color: rgb(248, 35, 117);overflow-wrap: inherit !important;word-break: inherit !important;"property/span>
6    <span class="hljs-tag" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">
7        <span class="hljs-tag" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">dfs.namenode.name.dir</span class="hljs-name" style="font-size: inherit;line-height: inherit;color: rgb(248, 35, 117);overflow-wrap: inherit !important;word-break: inherit !important;"name/span>
8        <span class="hljs-tag" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">file:/usr/local/hadoop/tmp/dfs/name</span class="hljs-name" style="font-size: inherit;line-height: inherit;color: rgb(248, 35, 117);overflow-wrap: inherit !important;word-break: inherit !important;"value/span>
9    </span class="hljs-name" style="font-size: inherit;line-height: inherit;color: rgb(248, 35, 117);overflow-wrap: inherit !important;word-break: inherit !important;"property/span>
10    <span class="hljs-tag" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">
11        <span class="hljs-tag" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">dfs.datanode.data.dir</span class="hljs-name" style="font-size: inherit;line-height: inherit;color: rgb(248, 35, 117);overflow-wrap: inherit !important;word-break: inherit !important;"name/span>
12        <span class="hljs-tag" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">file:/usr/local/hadoop/tmp/dfs/data</span class="hljs-name" style="font-size: inherit;line-height: inherit;color: rgb(248, 35, 117);overflow-wrap: inherit !important;word-break: inherit !important;"value/span>
13    </span class="hljs-name" style="font-size: inherit;line-height: inherit;color: rgb(248, 35, 117);overflow-wrap: inherit !important;word-break: inherit !important;"property/span>
14</span class="hljs-name" style="font-size: inherit;line-height: inherit;color: rgb(248, 35, 117);overflow-wrap: inherit !important;word-break: inherit !important;"configuration/span>
6 格式化 NameNode

/usr/local/hadoop/bin/hdfs namenode -format
在输出信息中看到如下信息，则表现格式化乐成:
1Storage directory /usr/local/hadoop/tmp/dfs/name has been successfully formatted.
2Exiting with status 0
7 启动 NameNode 和 DataNode 保卫历程

/usr/local/hadoop/sbin/start-dfs.sh
8 查抄NameNode和DataNode是否正常启动

1jps
2[root@gosaint ~]$ jps
33689 SecondaryNameNode
43520 DataNode
53800 Jps
63393 NameNode
访问：http://ip:50070/

9 关闭主步伐

末了，假如利用完毕，可以关闭Hadoop。
[root@centos7 hadoop-2.7.3]# sbin/stop-dfs.sh
10 YARN来提交job

设置1 mapred-site.xml
1修改模板文件名：
2mv /usr/local/hadoop/etc/hadoopmapred-site.xml.template /usr/local/hadoop/etc/hadoop/mapred-site.xml
1vi /usr/local/hadoop/etc/hadoop/mapred-site.xml
2
3<span class="hljs-tag" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">
4    <span class="hljs-tag" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">
5        <span class="hljs-tag" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">mapreduce.framework.name</span class="hljs-name" style="font-size: inherit;line-height: inherit;color: rgb(248, 35, 117);overflow-wrap: inherit !important;word-break: inherit !important;"name/span>
6        <span class="hljs-tag" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">yarn</span class="hljs-name" style="font-size: inherit;line-height: inherit;color: rgb(248, 35, 117);overflow-wrap: inherit !important;word-break: inherit !important;"value/span>
7    </span class="hljs-name" style="font-size: inherit;line-height: inherit;color: rgb(248, 35, 117);overflow-wrap: inherit !important;word-break: inherit !important;"property/span>
8</span class="hljs-name" style="font-size: inherit;line-height: inherit;color: rgb(248, 35, 117);overflow-wrap: inherit !important;word-break: inherit !important;"configuration/span>
设置2 设置yarn-site.xml
1<span class="hljs-tag" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">
2
3    <span class="hljs-tag" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">
4        <span class="hljs-tag" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">yarn.nodemanager.aux-services</span class="hljs-name" style="font-size: inherit;line-height: inherit;color: rgb(248, 35, 117);overflow-wrap: inherit !important;word-break: inherit !important;"name/span>
5        <span class="hljs-tag" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">mapreduce_shuffle</span class="hljs-name" style="font-size: inherit;line-height: inherit;color: rgb(248, 35, 117);overflow-wrap: inherit !important;word-break: inherit !important;"value/span>
6    </span class="hljs-name" style="font-size: inherit;line-height: inherit;color: rgb(248, 35, 117);overflow-wrap: inherit !important;word-break: inherit !important;"property/span>
7</span class="hljs-name" style="font-size: inherit;line-height: inherit;color: rgb(248, 35, 117);overflow-wrap: inherit !important;word-break: inherit !important;"configuration/span>
11：启动ResourceManager和NodeManager

留意：实行下面的下令之前，先确保已实行"sbin/start-dfs.sh"。
[root@centos7 hadoop-2.7.3]# sbin/start-yarn.sh
12：启动historyserver

[root@centos7 hadoop-2.7.3]# sbin/mr-jobhistory-daemon.sh start
确认历程已启动
1[root@gosaint ~]# jps
21670 ResourceManager
31272 NameNode
41769 NodeManager
51370 DataNode
62234 Jps
71501 SecondaryNameNode
81838 JobHistoryServer
访问：http://ip:8088/

至此，hadoop的伪分布式情况搭建完毕！