hadoop 手动安装操作

发表于 | 分类于

本地运行模式

详细参考 hadoop 部署一章

验证

  1. 创建文件

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    su - hadoop
    cd /opt/bigdata/hadoop/current
    mkdir wcinput

    cat > /opt/bigdata/hadoop/current/wcinput/demo.txt <<-EOF
    Linux Unix windows
    hadoop Linux spark
    hive hadoop Unix
    MapReduce hadoop  Linux hive
    windows hadoop spark
    EOF

  2. 执行程序

    1
    2
    cd /opt/bigdata/hadoop/current
    hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-*.jar wordcount wcinput wcoutput

  3. 查看结果

1
2
3
4
5
6
7
8
9
10
11
# ls wcoutput/
part-r-00000  _SUCCESS

# cat wcoutput/part-r-00000
Linux   3
MapReduce       1
Unix    2
hadoop  4
hive    2
spark   2
windows 2

备注

1
2
3
_SUCCESS : 任务完成标识，表示执行成功；

part-r-00000 :  m 标识的文件是 mapper 输出，带 r 标识的文件是 reduce 输出的，00000 为 job 任务编号，part-r-00000 整个文件为结果输出文件

单节点伪分布式示例

基本和完全分布式部署差不多，主要是配置文件有差异。先测试单节点伪分布式

节点 192.168.2.231，前置操作已经操作完。hosts 文件, 使用默认配置，仅作部分修改

192.168.2.231 hadoop231

/etc/hadoop/conf/core-site.xml, 改为当前节点，默认端口 rpc 8020

1
2
3
4
<property>
  <name>fs.defaultFS</name>
    <value>hdfs://hadoop231</value>
</property>

日志文件在 /opt/bigdata/hadoop/current/logs/ ,遇到问题可以看日志定位

启动 NameNode 服务

  1. 格式化 NameNode 

    1
    2
    3
    $ su - hadoop
    $ cd /opt/bigdata/hadoop/current/bin
    $ hdfs  namenode -format

  2. 启动 NameNode

1
$ hdfs --daemon start namenode
  1. 验证 
    1
    2
    $ jps|grep NameNode
    27319 NameNode

192.168.2.231:9870

NameNode 服务状态页面

hadoop01

datanode 还没有启动等，无数据

  1. 启动 secondarynamenode (分布式可以不需要这个服务)
    1
    2
    3
    $ hdfs --daemon start secondarynamenode
    $ jps|grep SecondaryNameNode
    29705 SecondaryNameNode

启动 DataNode 服务

  1. 启动 DataNode
1
hdfs --daemon start datanode
  1. 验证 
    1
    2
    $ jps|grep DataNode
    3876 DataNode

192.168.2.231:9870

NameNode 服务状态页面

hadoop02

可以看到已有活动的节点等

启动 ResourceManager 服务

  1. 启动 ResourceManager
1
$ yarn --daemon start resourcemanager
  1. 验证 
    1
    2
    $ jps|grep ResourceManager
    4726 ResourceManager

192.168.2.231:8088

ResourceManager 的状态页面

yarn界面

ResourceManager 中可用的内存资源、CPU 资源数及活跃节点数这些数据都是 0，是因为还没有 NodeManager 服务启动。

启动 NodeManager 服务

  1. 启动 ResourceManager

    1
    $ yarn --daemon start nodemanager

  2. 验证 

1
2
$ jps|grep NodeManager
8853 NodeManager

ResourceManager 的状态页面

yarn界面

启动 Jobhistoryserver 服务

  1. 启动 Jobhistoryserver

    1
    $ mapred  --daemon start historyserver

  2. 验证 

    1
    2
    $ jps|grep JobHistoryServer
    1027 JobHistoryServer

192.168.2.231:19888 JobHistoryServer
JobHistory

以上设置为最简配置

默认 mapreduce 的运行环境是 local（本地），要让 mapreduce 在 yarn 上运行, 需要修改

  1. 修改文件
    mapred-site.xml:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
<configuration>
<property>
   <name>mapreduce.framework.name</name>
   <value>yarn</value>
</property>

<property>
   <name>yarn.app.mapreduce.am.env</name>
   <value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
</property>

<property>
   <name>mapreduce.map.env</name>
   <value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
</property>

<property>
   <name>mapreduce.reduce.env</name>
   <value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
</property>
</configuration>

yarn-site.xml :

1
2
3
4
<property>
  <name>yarn.nodemanager.aux-services</name>
  <value>mapreduce_shuffle</value>
</property>
  1. 重启服务 ResourceManager 与 nodemanager 
1
2
3
4
5
yarn --daemon stop  resourcemanager
yarn --daemon stop  nodemanager

yarn --daemon start  resourcemanager
yarn --daemon start  nodemanager

测试

1
2
3
4
5
6
7
8
9
10
11
12
13
cat > /tmp/demo.txt <<-EOF
Linux Unix windows
hadoop Linux spark
hive hadoop Unix
MapReduce hadoop  Linux hive
windows hadoop spark
EOF

hadoop fs -mkdir /demo
hadoop fs -put /tmp/demo.txt /demo
hadoop jar /opt/bigdata/hadoop/current/share/hadoop/mapreduce/hadoop-mapreduce-examples-*.jar  wordcount /demo  /output

hadoop fs -ls /output

结果

1
2
3
4
5
6
7
8
$ hadoop fs -text /output/part-r-00000
Linux   3
MapReduce       1
Unix    2
hadoop  4
hive    2
spark   2
windows 2

有历史记录

遇到的问题

namenode 与 datanode 冲突

服务器重启后，重新 hdfs namenode -format 导致 version 不一致
hadoop问题1

hadoop问题定位1

备份后 删除 data tmp 文件

mapreduce 改为集群后, yarn-site 当时未修改

hadoop问题定位2

改完后，重启重跑任务正常

问题2

完全分布式运行模式

前置操作

所有节点均先完成 前置操作，hadoop 原生安装 (本地运行模式) 的操作。详细参考 hadoop 部署

当前 /etc/hosts

1
2
3
4
5
6
7
8
cat >/etc/hosts<<eof
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

192.168.2.241  hadoop01
192.168.2.242  hadoop02
192.168.2.243  hadoop03
eof

节点规划

一般情况下:

  1. NameNode 服务要独立部署
  2. datanode 和 nodemanager 服务建议部署在一台服务器上
  3. resourcemanager 服务要独立部署
  4. historyserver 一般和 resourcemanager 部署在一起
  5. ZooKeeper (QuorumPeerMain) 和 JournalNode 集群可以放在一起
  6. zkfc (DFSZKFailoverController) 是对 NameNode 进行资源仲裁的，所以它必须和 NameNode 服务运行在一起

当前资源限制, 只有三台节点，做如下规划
hadoop01 : QuorumPeerMain, JournalNode, NameNode, DFSZKFailoverController, DataNode, NodeManager
hdaoop02 : QuorumPeerMain, JournalNode, NameNode, DFSZKFailoverController, DataNode, NodeManager
hadoop03 : QuorumPeerMain, JournalNode, DataNode, ResourceManager, NodeManager, JobHistoryServer

安装 zookeeper

官方网站 https://zookeeper.apache.org/releases.html#download

所有节点执行

1
2
3
4
5
6
wget --no-check-certificate https://dlcdn.apache.org/zookeeper/zookeeper-3.8.0/apache-zookeeper-3.8.0-bin.tar.gz

mkdir -p /opt/bigdata/zookeeper
tar -zxvf apache-zookeeper-*-bin.tar.gz -C /opt/bigdata/zookeeper
cd /opt/bigdata/zookeeper
ln -s apache-zookeeper-*-bin current

所有节点执行 修改配置文件及创建文件夹
/opt/bigdata/zookeeper/current/conf/zoo.cfg

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
cat > /opt/bigdata/zookeeper/current/conf/zoo.cfg <<-EOF
tickTime=2000
initLimit=20
syncLimit=10
dataDir=/opt/bigdata/zookeeper/current/data
dataLogDir=/opt/bigdata/zookeeper/current/dataLogDir
clientPort=2181
quorumListenOnAllIPs=true
server.1=hadoop01:2888:3888
server.2=hadoop02:2888:3888
server.3=hadoop03:2888:3888
admin.serverPort=8081
EOF

mkdir -p /opt/bigdata/zookeeper/current/data
mkdir -p /opt/bigdata/zookeeper/current/dataLogDir

  1. 2181 : 对 client 端提供服务
  2. 2888 : 集群内机器通信使用
  3. 3888 : 选举 leader 使用

写入 myid ，每个节点不同

1
2
3
echo 1 > /opt/bigdata/zookeeper/current/data/myid # hdp01
echo 2 > /opt/bigdata/zookeeper/current/data/myid # hdp02
echo 3 > /opt/bigdata/zookeeper/current/data/myid # hdp03

所有节点执行授权

1
chown -R hadoop:hadoop /opt/bigdata/zookeeper

启动 zookeeper

1
2
3
4
5
6
7
$ cd /opt/bigdata/zookeeper/current/bin

$ ./zkServer.sh  start
$ jps
23097 QuorumPeerMain

$ ./zkServer.sh  status

日志文件目录 /opt/bigdata/zookeeper/current/logs

修改配置文件及相关操作

配置文件路径 /etc/hadoop/conf ，所有节点均修改

修改配置文件

  1. core-site.yml

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    <configuration>

    <property>
      <name>fs.defaultFS</name>
      <value>hdfs://bigdata</value>
    </property>

    <property>
      <name>hadoop.tmp.dir</name>
      <value>/var/tmp/hadoop-${user.name}</value>
    </property>


    <property>
      <name>ha.zookeeper.quorum</name>
      <value>hadoop01,hadoop02,hadoop03</value>
    </property>

      <property>
        <name>fs.trash.interval</name>
        <value>60</value>
      </property>
    </configuration>

  2. hdfs-site.yml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
<configuration>
   <property>
        <name>dfs.nameservices</name>
        <value>bigdata</value>
    </property>

    <property>
        <name>dfs.ha.namenodes.bigdata</name>
        <value>nn1,nn2</value>
    </property>


    <property>
        <name>dfs.namenode.rpc-address.bigdata.nn1</name>
        <value>hadoop01:9000</value>
    </property>


    <property>
        <name>dfs.namenode.rpc-address.bigdata.nn2</name>
        <value>hadoop02:9000</value>
    </property>


    <property>
        <name>dfs.namenode.http-address.bigdata.nn1</name>
        <value>hadoop01:50070</value>
    </property>

    <property>
        <name>dfs.namenode.http-address.bigdata.nn2</name>
        <value>hadoop02:50070</value>
    </property>


    <property>
        <name>dfs.namenode.shared.edits.dir</name>
        <value>qjournal://hadoop01:8485;hadoop02:8485;hadoop03:8485/bigdata</value>
    </property>


<property>
        <name>dfs.ha.automatic-failover.enabled.bigdata</name>
        <value>true</value>
</property>

<property>
  <name>dfs.client.failover.proxy.provider.bigdata</name>
  <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
</property>


<property>
        <name>dfs.journalnode.edits.dir</name>
        <value>/data1/hadoop/dfs/jn</value>
</property>


<property>
        <name>dfs.replication</name>
        <value>2</value>
</property>


<property>
        <name>dfs.ha.fencing.methods</name>
        <value>shell(/bin/true)</value>
</property>


<property>
  <name>dfs.namenode.name.dir</name>
  <value>file:///data1/hadoop/dfs/name,file:///data2/hadoop/dfs/name</value>
  <final>true</final>
</property>


<property>
  <name>dfs.datanode.data.dir</name>
<value>file:///data1/hadoop/dfs/data,file:///data2/hadoop/dfs/data</value>
  <final>true</final>
</property>


<property>
  <name>dfs.block.size</name>
  <value>134217728</value>
</property>


<property>
  <name>dfs.permissions</name>
  <value>true</value>
</property>


<property>
  <name>dfs.permissions.supergroup</name>
  <value>supergroup</value>
</property>


<property>
  <name>dfs.hosts</name>
  <value>/etc/hadoop/conf/hosts</value>
</property>


<property>
  <name>dfs.hosts.exclude</name>
  <value>/etc/hadoop/conf/hosts-exclude</value>
</property>

</configuration>
  1. mapred-site.xml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
<configuration>

<property>
 <name>mapreduce.framework.name</name>
 <value>yarn</value>
</property>

<property>
 <name>mapreduce.jobhistory.address</name>
 <value>hadoop03:10020</value>
</property>

<property>
 <name>mapreduce.jobhistory.webapp.address</name>
 <value>hadoop03:19888</value>
</property>

</configuration>
  1. yarn-site.xml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
<configuration>

<property>
    <name>yarn.resourcemanager.hostname</name>
    <value>hadoop03</value>
</property>

<property>
    <name>yarn.resourcemanager.scheduler.address</name>
    <value>hadoop03:8030</value>
</property>

<property>
    <name>yarn.resourcemanager.resource-tracker.address</name>
    <value>hadoop03:8031</value>
</property>

<property>
    <name>yarn.resourcemanager.address</name>
    <value>hadoop03:8032</value>
</property>

<property>
    <name>yarn.resourcemanager.admin.address</name>
    <value>hadoop03:8033</value>
</property>

<property>
    <name>yarn.resourcemanager.webapp.address</name>
    <value>hadoop03:8088</value>
</property>

<property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
</property>

<property>
    <name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
    <value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>

<property>
      <name>yarn.nodemanager.aux-services.spark_shuffle.class</name>
      <value>org.apache.spark.network.yarn.YarnShuffleService</value>
</property>

<property>
    <name>yarn.nodemanager.local-dirs</name>
    <value>file:///data1/hadoop/yarn/local,file:///data2/hadoop/yarn/local</value>
</property>

<property>
    <name>yarn.nodemanager.log-dirs</name>
    <value>file:///data1/hadoop/yarn/logs,file:///data2/hadoop/yarn/logs</value>
</property>


<property>
    <description>Classpath for typical applications.</description>
     <name>yarn.application.classpath</name>
     <value>
        $HADOOP_CONF_DIR,
        $HADOOP_COMMON_HOME/*,$HADOOP_COMMON_HOME/lib/*,
        $HADOOP_HDFS_HOME/*,$HADOOP_HDFS_HOME/lib/*,
        $HADOOP_MAPRED_HOME/*,$HADOOP_MAPRED_HOME/lib/*,
        $HADOOP_YARN_HOME/*,$HADOOP_YARN_HOME/lib/*,
        $HADOOP_HOME/share/hadoop/common/*, $HADOOP_COMMON_HOME/share/hadoop/common/lib/*,
        $HADOOP_HOME/share/hadoop/hdfs/*, $HADOOP_HOME/share/hadoop/hdfs/lib/*,
        $HADOOP_HOME/share/hadoop/mapreduce/*, $HADOOP_HOME/share/hadoop/mapreduce/lib/*,
        $HADOOP_HOME/share/hadoop/yarn/*, $HADOOP_YARN_HOME/share/hadoop/yarn/lib/*,
        $HIVE_HOME/lib/*, $HIVE_HOME/lib_aux/*
     </value>
</property>

<property>
  <name>yarn.nodemanager.resource.memory-mb</name>
    <value>20480</value>
</property>

<property>
    <name>yarn.nodemanager.resource.cpu-vcores</name>
    <value>8</value>
</property>

</configuration>
  1. hosts 文件
1
2
3
4
5
cat >/etc/hadoop/conf/hosts<<-EOF
hadoop01
hadoop02
hadoop03
EOF
其他操作

根据配置文件创建目录， 所有节点

1
2
3
4
mkdir -p /data1/hadoop
mkdir -p /data2/hadoop
chown -R hadoop:hadoop /data1/hadoop
chown -R hadoop:hadoop /data2/hadoop

安装

均在 hadoop 用户下执行 su hadoop

  1. 启动与格式化 ZooKeeper 集群 (所有节点) 
1
2
3
4
$ cd /opt/bigdata/zookeeper/current/bin
$ /zkServer.sh  start
$ jps
2182 QuorumPeerMain
  1. ZooKeeper 格式化 (hadoop01)
1
hdfs zkfc -formatZK

  1. 启动 JournalNode 集群 (所有节点)，产生 /data1/hadoop/dfs/jn

    1
    2
    3
    $ hdfs --daemon start journalnode
    $ jps
    15187 JournalNode

  2. 格式化并启动主节点 NameNode 服务 (hadoop01)

1
2
3
4
$ hdfs namenode -format -clusterId bigdataserver  # 此名称可随便指定
$ hdfs --daemon start namenode
$ jps
11724 NameNode
  1. NameNode 主、备节点同步元数据,并启动备节点 (hadoop02)
1
2
3
4
$ hdfs namenode -bootstrapStandby
$ hdfs --daemon start namenode
$ jps
1880 NameNode

  1. 启动 ZooKeeper FailoverController（zkfc）服务 (hadoop01启动后，再启动 hadoop02)

    1
    2
    3
    $ hdfs --daemon start zkfc
    $ jps
    1888 DFSZKFailoverController

  2. 启动存储节点 DataNode 服务 (所有节点)

1
2
3
$ hdfs --daemon start datanode
$ jps
2880 DataNode
  1. 启动 ResourceManager、NodeManager 及 historyserver 服务
1
2
3
4
5
6
7
$ yarn --daemon start resourcemanager  # hadoop03 
$ yarn --daemon start nodemanager  # 所有节点
$ mapred --daemon start historyserver # hadoop03
$ jps
7986 ResourceManager
8246 NodeManager
8405 JobHistoryServer

验证

上传文件及 mapreduce

1
2
3
4
5
6
7
8
9
10
11
12
13
cat > /tmp/demo.txt <<-EOF
Linux Unix windows
hadoop Linux spark
hive hadoop Unix
MapReduce hadoop  Linux hive
windows hadoop spark
EOF

hadoop fs -mkdir /demo
hadoop fs -put /tmp/demo.txt /demo
hadoop jar /opt/bigdata/hadoop/current/share/hadoop/mapreduce/hadoop-mapreduce-examples-*.jar  wordcount /demo  /output

hadoop fs -ls /output

结果

1
2
3
4
5
6
7
8
$ hadoop fs -text /output/part-r-00000
Linux   3
MapReduce       1
Unix    2
hadoop  4
hive    2
spark   2
windows 2

网页

主namenode

备namenode

cluster_node

提交的任务