未加星标

CentOS 7 安装Hadoop 2.7.1

字体大小 | |
[系统(linux) 所属分类 系统(linux) | 发布者 店小二04 | 时间 20160901 | 作者 红领巾 ] 0人收藏点击收藏
两台机器 CentOS7(机器名分别为master-CentOS7、slave-CentOS7) 内存2G (笔记本开虚拟机快撑不住了╮(╯-╰)╭
CentOS7 与 CetnOS6 有一些区别
网络配置

master-CentOS7

[[email protected] ~]# vi /etc/sysconfig/network-scripts/ifcfg-eno16777736
TYPE=Ethernet
BOOTPROTO=static
DEFROUTE=yes
PEERDNS=yes
PEERROUTES=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_PEERDNS=yes
IPV6_PEERROUTES=yes
IPV6_FAILURE_FATAL=no
NAME=eno16777736
UUID=b30f5765-ecd7-4dba-a0ed-ebac92c836bd
DEVICE=eno16777736
ONBOOT=yes
IPADDR=192.168.1.182
NETMASK=255.255.255.0
GATEWAY=192.168.1.1
DNS1=114.114.114.114
DNS2=8.8.4.4
网络信息根据自己实际的网络情况配置。
[[email protected] ~]# systemctl restart network
[[email protected] ~]# ifconfig

slave-CentOS7

[[email protected] ~]# vi /etc/sysconfig/network-scripts/ifcfg-eno16777736
TYPE=Ethernet
BOOTPROTO=static
DEFROUTE=yes
PEERDNS=yes
PEERROUTES=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_PEERDNS=yes
IPV6_PEERROUTES=yes
IPV6_FAILURE_FATAL=no
NAME=eno16777736
UUID=b30f5765-ecd7-4dba-a0ed-ebac92c836bd
DEVICE=eno16777736
ONBOOT=yes
IPADDR=192.168.1.183
NETMASK=255.255.255.0
GATEWAY=192.168.1.1
DNS1=114.114.114.114
DNS2=8.8.4.4
网络信息根据自己实际的网络情况配置。
[[email protected] ~]# systemctl restart network
[[email protected] ~]# ifconfig
设置hosts、hostname

master-CentOS7

[[email protected] ~]# vi /etc/hosts
添加
192.168.1.182 master
192.168.1.183 slave
[[email protected] ~]# vi /etc/hostname
localhost.localdomain
内容修改为
master

slave-CentOS7

[[email protected] ~]# vi /etc/hosts
添加
192.168.1.182 master
192.168.1.183 slave
[[email protected] ~]# vi /etc/hostname
localhost.localdomain
内容修改为
slave
关闭selinux

master-CentOS7

[[email protected] ~]# getenforce
Enforcing
[[email protected] ~]# vi /etc/selinux/config
SELINUX=enforcing
修改为
SELINUX=disabled
保存重启
[[email protected] ~]# getenforce
Disabled

slave-CentOS7

[[email protected] ~]# getenforce
Enforcing
[[email protected] ~]# vi /etc/selinux/config
SELINUX=enforcing
修改为
SELINUX=disabled
保存重启
[[email protected] ~]# getenforce
Disabled
关闭firewalld

master-CentOS7

[[email protected] ~]# systemctl disable firewalld
Removed symlink /etc/systemd/system/dbus-org.Fedoraproject.FirewallD1.service.
Removed symlink /etc/systemd/system/basic.target.wants/firewalld.service.
[[email protected] ~]# systemctl stop firewalld
[[email protected] ~]# iptables -nvL
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
[[email protected] ~]# yum install -y iptables-services
[[email protected] ~]# service iptables save
iptables: Saving firewall rules to /etc/sysconfig/iptables:[ 确定 ]
[[email protected] ~]# systemctl enable iptables
Created symlink from /etc/systemd/system/basic.target.wants/iptables.service to /usr/lib/systemd/system/iptables.service.

slave-CentOS7

[[email protected] ~]# systemctl disable firewalld
Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.
Removed symlink /etc/systemd/system/basic.target.wants/firewalld.service.
[[email protected] ~]# systemctl stop firewalld
[[email protected] ~]# iptables -nvL
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
[[email protected] ~]# yum install -y iptables-services
[[email protected] ~]# service iptables save
iptables: Saving firewall rules to /etc/sysconfig/iptables:[ 确定 ]
[[email protected] ~]# systemctl enable iptables
Created symlink from /etc/systemd/system/basic.target.wants/iptables.service to /usr/lib/systemd/system/iptables.service.
密钥登陆

master-CentOS7

[[email protected] ~]# ssh-keygen
一直回车
[[email protected] ~]# cat .ssh/id_rsa.pub
复制~/.ssh/id_rsa.pub 内容

slave-CentOS7

[[email protected] ~]# vi .ssh/authorized_keys
复制~/.ssh/id_rsa.pub 内容到 ~/.ssh/authorized_keys时报错
".ssh/authorized_keys" E212: Can't open file for writing
解决方案
[[email protected] ~]# ls -ld .ssh
ls: 无法访问.ssh: 没有那个文件或目录
[[email protected] ~]# mkdir .ssh; chmod 700 .ssh
[[email protected] ~]# ls -ld .ssh
drwx------ 2 root root 6 8月 28 15:59 .ssh
[[email protected] ~]# vi .ssh/authorized_keys
复制~/.ssh/id_rsa.pub 内容到 ~/.ssh/authorized_keys
[[email protected] ~]# ls -l !$
ls -l .ssh/authorized_keys
-rw-r--r-- 1 root root 418 8月 28 16:02 .ssh/authorized_keys

master-CentOS7

[[email protected] ~]# vi .ssh/authorized_keys
复制~/.ssh/id_rsa.pub 内容到 ~/.ssh/authorized_keys

测试

master-CentOS7

[[email protected] ~]# ssh master
[[email protected] ~]# exit
[[email protected] ~]# ssh slave
[[email protected] ~]# exit
安装JDK

Hadoop2.7 需要安装jdk1.7版本,下载地址http://www.oracle.com/technetwork/java/javase/downloads/jdk7-downloads-1880260.html

先卸载CetnOS7自带的JDK
以slave-CentOS7为例(master-CetnOS7、slave-CentOS7上都需要卸载CetnOS7自带的JDK)
[[email protected] ~]# java -version
openjdk version "1.8.0_101"
OpenJDK Runtime Environment (build 1.8.0_101-b13)
OpenJDK 64-Bit Server VM (build 25.101-b13, mixed mode)
[[email protected] ~]# rpm -qa |grep jdk
java-1.7.0-openjdk-headless-1.7.0.111-2.6.7.2.el7_2.x86_64
java-1.8.0-openjdk-1.8.0.101-3.b13.el7_2.x86_64
java-1.8.0-openjdk-headless-1.8.0.101-3.b13.el7_2.x86_64
java-1.7.0-openjdk-1.7.0.111-2.6.7.2.el7_2.x86_64
[[email protected] ~]# yum -y remove java-1.7.0-openjdk-headless-1.7.0.111-2.6.7.2.el7_2.x86_64
[[email protected] ~]# yum -y remove java-1.8.0-openjdk-1.8.0.101-3.b13.el7_2.x86_64
[[email protected] ~]# yum -y remove java-1.8.0-openjdk-headless-1.8.0.101-3.b13.el7_2.x86_64
[[email protected] ~]# java -version
-bash: /usr/bin/java: 没有那个文件或目录

master-CentOS7

[[email protected] ~]# wget 'http://download.Oracle.com/otn-pub/java/jdk/7u79-b15/jdk-7u79-linux-x64.tar.gz?AuthParam=1472372876_f3205a608139acb432d3c48638502428'
[[email protected] ~]# mv jdk-7u79-linux-x64.tar.gz\?AuthParam\=1472372876_f3205a608139acb432d3c48638502428 jdk-7u79-linux-x64.tar.gz
[[email protected] ~]# tar zxvf jdk-7u79-linux-x64.tar.gz
[[email protected] ~]# mv jdk1.7.0_79 /usr/local/
[[email protected] ~]# vi /etc/profile.d/java.sh
添加
export JAVA_HOME=/usr/local/jdk1.7.0_79
export CLASSPATH=.:$JAVA_HOME/jre/lib/rt.jar:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export PATH=$PATH:$JAVA_HOME/bin
[[email protected] ~]# source !$
source /etc/profile.d/java.sh
[[email protected] ~]# java -version
java version "1.7.0_79"
Java(TM) SE Runtime Environment (build 1.7.0_79-b15)
Java HotSpot(TM) 64-Bit Server VM (build 24.79-b02, mixed mode)
[[email protected] ~]# scp jdk-7u79-linux-x64.tar.gz slave:/root/
[[email protected] ~]# scp /etc/profile.d/java.sh slave:/etc/profile.d/

slave-CentOS7

[[email protected] ~]# tar zxvf jdk-7u79-linux-x64.tar.gz
[[email protected] ~]# mv jdk1.7.0_79 /usr/local/
[[email protected] ~]# source /etc/profile.d/java.sh
[[email protected] ~]# java -version
java version "1.7.0_79"
Java(TM) SE Runtime Environment (build 1.7.0_79-b15)
Java HotSpot(TM) 64-Bit Server VM (build 24.79-b02, mixed mode)
安装Hadoop

master-CentOS7

[[email protected] ~]# wget 'http://mirror.bit.edu.cn/apache/hadoop/common/hadoop-2.7.1/hadoop-2.7.1.tar.gz'
[[email protected] ~]# tar zxvf hadoop-2.7.1.tar.gz
[[email protected] ~]# mv hadoop-2.7.1 /usr/local/Hadoop
[[email protected] ~]# ls !$
ls /usr/local/hadoop
bin include libexec NOTICE.txt sbin
etc lib LICENSE.txt README.txt share
[[email protected] ~]# mkdir /usr/local/hadoop/tmp /usr/local/hadoop/dfs /usr/local/hadoop/dfs/data /usr/local/hadoop/dfs/name
[[email protected] ~]# ls /usr/local/hadoop
bin dfs etc include lib libexec LICENSE.txt NOTICE.txt README.txt sbin share tmp
[[email protected] ~]# rsync -av /usr/local/hadoop slave:/usr/local

slave-CentOS7

[[email protected] ~]# ls /usr/local/hadoop
bin etc lib LICENSE.txt README.txt share
dfs include libexec NOTICE.txt sbin tmp
配置Hadoop

master-CentOS7

[[email protected] ~]# vi /usr/local/hadoop/etc/hadoop/core-site.xml
添加
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://192.168.1.182:9000</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>file:/usr/local/hadoop/tmp</value>
</property>
<property>
<name>io.file.buffer.size</name>
<value>131702</value>
</property>
</configuration>
注意master-CentOS7主机的IP
[[email protected] ~]# vi /usr/local/hadoop/etc/hadoop/hdfs-site.xml
添加
<configuration>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:/usr/local/hadoop/dfs/name</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:/usr/local/hadoop/dfs/data</value>
</property>
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>192.168.1.182:9001</value>
</property>
<property>
<name>dfs.webhdfs.enabled</name>
<value>true</value>
</property>
</configuration>
注意master-CentOS7主机的IP
[[email protected] ~]# mv /usr/local/hadoop/etc/hadoop/mapred-site.xml.template /usr/local/hadoop/etc/hadoop/mapred-site.xml
[[email protected] ~]# vi /usr/local/hadoop/etc/hadoop/mapred-site.xml
添加
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>192.168.1.182:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>192.168.1.182:19888</value>
</property>
</configuration>
注意master-CentOS7主机的IP
[[email protected] ~]# vi /usr/local/hadoop/etc/hadoop/yarn-site.xml
添加
<configuration>
<!-- Site specific YARN configuration properties -->
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.auxservices.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>192.168.1.182:8032</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>192.168.1.182:8030</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>192.168.1.182:8031</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address</name>
<value>192.168.1.182:8033</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>192.168.1.182:8088</value>
</property>
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>2048</value>
</property>
</configuration>
注意master-CentOS7主机的IP
[[email protected] ~]# cd /usr/local/hadoop/etc/hadoop
[[email protected] hadoop]# vi hadoop-env.sh
更改
export JAVA_HOME=/usr/local/jdk1.7.0_79
[[email protected] hadoop]# vi yarn-env.sh
更改
export JAVA_HOME=/usr/local/jdk1.7.0_79
[[email protected] hadoop]# vi slaves
更改为
192.168.1.183
注意slave-CentOS7的IP
[[email protected] hadoop]# rsync -av /usr/local/hadoop/etc/ slave:/usr/local/hadoop/etc/

slave-CentOS7

[[email protected] ~]# cd /usr/local/hadoop/etc/hadoop/
[[email protected] hadoop]# cat slaves
192.168.1.183
检查slave没问题
启动Hadoop

master-CentOS7

[[email protected] hadoop]# /usr/local/hadoop/bin/hdfs namenode -format
[[email protected] hadoop]# echo $?
0
[[email protected] hadoop]# /usr/local/hadoop/sbin/start-all.sh
[[email protected] hadoop]# jps
19907 ResourceManager
19604 SecondaryNameNode
19268 NameNode
20323 Jps

slave-CentOS7

[[email protected] hadoop]# jps
18113 NodeManager
18509 Jps
17849 DataNode
浏览器打开
http://192.168.1.182:8088/
http://192.168.1.182:50070/
测试Hadoop

master-CentOS7

[[email protected] hadoop]# cd /usr/local/hadoop/
[[email protected] hadoop]# bin/hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.1.jar pi 10 10
停止服务

master-CentOS7

[[email protected] hadoop]# cd /usr/local/hadoop
[[email protected] hadoop]# sbin/stop-all.sh
如果提示 copyFromLocal: Cannot create directory /123/. Name node is in safe mode.
这是因为开启了安全模式
解决方法:
cd /usr/local/Hadoop
bin/hdfs dfsadmin -safemode leave

更多Hadoop相关信息见Hadoop 专题页面 http://www.codesec.net/topicnews.aspx?tid=13

本文地址:http://www.codesec.net/Linux/2016-09/135522.htm


CentOS 7 安装Hadoop 2.7.1

本文系统(linux)相关术语:linux系统 鸟哥的linux私房菜 linux命令大全 linux操作系统

主题: HadoopJavaUTLinux笔记本浏览器TIADMAUTAU
分页:12
转载请注明
本文标题:CentOS 7 安装Hadoop 2.7.1
本站链接:http://www.codesec.net/view/481858.html
分享请点击:


1.凡CodeSecTeam转载的文章,均出自其它媒体或其他官网介绍,目的在于传递更多的信息,并不代表本站赞同其观点和其真实性负责;
2.转载的文章仅代表原创作者观点,与本站无关。其原创性以及文中陈述文字和内容未经本站证实,本站对该文以及其中全部或者部分内容、文字的真实性、完整性、及时性,不作出任何保证或承若;
3.如本站转载稿涉及版权等问题,请作者及时联系本站,我们会及时处理。
登录后可拥有收藏文章、关注作者等权限...
技术大类 技术大类 | 系统(linux) | 评论(0) | 阅读(38)