未加星标

Ubuntu 14.04 LTS下安装Hadoop 1.2.1(伪分布模式)

字体大小 | |
[系统(linux) 所属分类 系统(linux) | 发布者 店小二04 | 时间 20160901 | 作者 红领巾 ] 0人收藏点击收藏

Hadoop的运行模式可分为单机模式、伪分布模式和分布模式。首先无论哪种模式都需要安装JDK的,这一步之前的随笔Ubuntu 14.04 LTS下安装JDK 1.8中已经做了(见 http://www.linuxidc.com/Linux/2016-09/135403.htm)。这里就不多说了。

其次是安装SSH。安装SSH是为了每次可以免密码登陆数据节点服务器。因为集群的环境下,每次登陆到数据节点服务器不可能每次都输入密码。这一步在前面的随笔Ubuntu 14.04 LTS下配置SSH免密码登录中已经做了(见 http://www.codesec.net/Linux/2016-09/135404.htm)。这里也不多说了。

伪分布模式安装:

首先下载Hadoop 1.2.1到本机,再解压到用户目录下。

[email protected]:~/Downloads$ tar zxf hadoop-1.2.1.tar.gz -C ~/hadoop_1.2.1
[email protected]:~/Downloads$ cd ~/hadoop_1.2.1/
[email protected]:~/hadoop_1.2.1$ ls
hadoop-1.2.1
[email protected]:~/hadoop_1.2.1$ cd hadoop-1.2.1/
[email protected]:~/hadoop_1.2.1/hadoop-1.2.1$ ls
bin hadoop-ant-1.2.1.jar ivy sbin
build.xml hadoop-client-1.2.1.jar ivy.xml share
c++ hadoop-core-1.2.1.jar lib src
CHANGES.txt hadoop-examples-1.2.1.jar libexec webapps
conf hadoop-minicluster-1.2.1.jar LICENSE.txt
contrib hadoop-test-1.2.1.jar NOTICE.txt
docs hadoop-tools-1.2.1.jar README.txt

然后配置hadoop的几个配置文件,都是XML格式。

首先是core-default.xml。这里配置hadoop分布式文件系统的地址和端口,以及Hadoop临时文件目录(/tmp/hadoop-${user.name})。

[email protected]:~/hadoop_1.2.1/hadoop-1.2.1/conf$ cat core-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://localhost:9000</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/hadoop/hadooptmp</value>
</property>
</configuration>
[email protected]:~/hadoop_1.2.1/hadoop-1.2.1/conf$

修改hadoop系统环境配置文件,告诉hadoop安装好的jdk的主目录路径

[email protected]:~/hadoop_1.2.1/hadoop-1.2.1$ cd conf/
[email protected]:~/hadoop_1.2.1/hadoop-1.2.1/conf$ ls
capacity-scheduler.xml hadoop-policy.xml slaves
configuration.xsl hdfs-site.xml ssl-client.xml.example
core-site.xml log4j.properties ssl-server.xml.example
fair-scheduler.xml mapred-queue-acls.xml taskcontroller.cfg
hadoop-env.sh mapred-site.xml task-log4j.properties
hadoop-metrics2.properties masters
[email protected]:~/hadoop_1.2.1/hadoop-1.2.1/conf$ sudo vim hadoop-env.sh n
[sudo] password for linuxidc:
2 files to edit
[email protected]:~/hadoop_1.2.1/hadoop-1.2.1/conf$ sudo vim hadoop-env.sh
[email protected]:~/hadoop_1.2.1/hadoop-1.2.1/conf$ tail -n 1 hadoop-env.sh
export JAVA_HOME=/usr/lib/jvm/jdk

然后是hdfs-site.xml 。修改hdfs的文件备份数量为1,dfs命名节点的主目录,dfs数据节点的目录。

[email protected]:~/hadoop_1.2.1/hadoop-1.2.1/conf$ cat hdfs-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.name.dir</name>
<value>/hadoop/hdfs/name</value>
</property>
<property>
<name>dfs.data.dir</name>
<value>/hadoop/hdfs/data</value>
</property>
</configuration>
[email protected]:~/hadoop_1.2.1/hadoop-1.2.1/conf$

最后配置mapreduce的job tracker的地址和端口

[email protected]:~/hadoop_1.2.1/hadoop-1.2.1/conf$ cat mapred-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>
<property>
<name>mapred.job.tracker</name>
<value>localhost:9001</value>
</property>
</configuration>
[email protected]:~/hadoop_1.2.1/hadoop-1.2.1/conf$

配置masters文件和slaves文件,这里因为我们是伪分布式,命名节点和数据节点其实都是一样。

[email protected]:~/hadoop_1.2.1/hadoop-1.2.1/conf$ cat masters
localhost
192.168.2.100
[email protected]:~/hadoop_1.2.1/hadoop-1.2.1/conf$ cat slaves
localhost
192.168.2.100
[email protected]:~/hadoop_1.2.1/hadoop-1.2.1/conf$

编辑/etc/hosts文件,配置主机名和IP地址的映射关系

[email protected]:~/hadoop_1.2.1/hadoop-1.2.1/conf$ cat /etc/hosts
127.0.0.1 localhost
127.0.1.1 ubuntu
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
192.168.2.100 master
192.168.2.100 slave
[email protected]:~/hadoop_1.2.1/hadoop-1.2.1/conf$

创建好core-default.xml,hdfs-site.xml,mapred-site.xml 三个配置文件里面写到的目录

[email protected]:~/hadoop_1.2.1/hadoop-1.2.1/conf$ mkdir -p /hadoop/hadooptmp
[email protected]:~/hadoop_1.2.1/hadoop-1.2.1/conf$ mkdir -p /hadoop/hdfs/name
[email protected]:~/hadoop_1.2.1/hadoop-1.2.1/conf$ mkdir -p /hadoop/hdfs/data

格式化HDFS

[email protected]:~/hadoop_1.2.1/hadoop-1.2.1/bin$ ./hadoop namenode -format

启动所有Hadoop服务,包括JobTracker,TaskTracker,Namenode

[email protected]:~/hadoop_1.2.1/hadoop-1.2.1/bin$ ./start-all.sh
starting namenode, logging to /home/linuxidc/hadoop_1.2.1/hadoop-1.2.1/libexec/../logs/hadoop-linuxidc-namenode-ubuntu.out
192.168.68.130: starting datanode, logging to /home/linuxidc/hadoop_1.2.1/hadoop-1.2.1/libexec/../logs/hadoop-linuxidc-datanode-ubuntu.out
localhost: starting datanode, logging to /home/linuxidc/hadoop_1.2.1/hadoop-1.2.1/libexec/../logs/hadoop-linuxidc-datanode-ubuntu.out
localhost: ulimit -a for user linuxidc
localhost: core file size (blocks, -c) 0
localhost: data seg size (kbytes, -d) unlimited
localhost: scheduling priority (-e) 0
localhost: file size (blocks, -f) unlimited
localhost: pending signals (-i) 7855
localhost: max locked memory (kbytes, -l) 64
localhost: max memory size (kbytes, -m) unlimited
localhost: open files (-n) 1024
localhost: pipe size (512 bytes, -p) 8
localhost: starting secondarynamenode, logging to /home/linuxidc/hadoop_1.2.1/hadoop-1.2.1/libexec/../logs/hadoop-linuxidc-secondarynamenode-ubuntu.out
192.168.68.130: secondarynamenode running as process 10689. Stop it first.
starting jobtracker, logging to /home/linuxidc/hadoop_1.2.1/hadoop-1.2.1/libexec/../logs/hadoop-linuxidc-jobtracker-ubuntu.out
192.168.68.130: starting tasktracker, logging to /home/linuxidc/hadoop_1.2.1/hadoop-1.2.1/libexec/../logs/hadoop-linuxidc-tasktracker-ubuntu.out
localhost: starting tasktracker, logging to /home/linuxidc/hadoop_1.2.1/hadoop-1.2.1/libexec/../logs/hadoop-linuxidc-tasktracker-ubuntu.out
localhost: ulimit -a for user linuxidc
localhost: core file size (blocks, -c) 0
localhost: data seg size (kbytes, -d) unlimited
localhost: scheduling priority (-e) 0
localhost: file size (blocks, -f) unlimited
localhost: pending signals (-i) 7855
localhost: max locked memory (kbytes, -l) 64
localhost: max memory size (kbytes, -m) unlimited
localhost: open files (-n) 1024
localhost: pipe size (512 bytes, -p) 8
[email protected]:~/hadoop_1.2.1/hadoop-1.2.1/bin$

查看Hadoop服务是否启动成功

[email protected]:~/hadoop_1.2.1/hadoop-1.2.1/conf$ jps
3472 JobTracker
3604 TaskTracker
3084 NameNode
5550 Jps
3247 DataNode
3391 SecondaryNameNode
[email protected]:~/hadoop_1.2.1/hadoop-1.2.1/conf$

查看hadoop群集的状态

[email protected]:~/hadoop_1.2.1/hadoop-1.2.1/bin$ ./hadoop dfsadmin -report
Configured Capacity: 41083600896 (38.26 GB)
Present Capacity: 32723169280 (30.48 GB)
DFS Remaining: 32723128320 (30.48 GB)
DFS Used: 40960 (40 KB)
DFS Used%: 0%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
-------------------------------------------------
Datanodes available: 1 (1 total, 0 dead)
Name: 127.0.0.1:50010
Decommission Status : Normal
Configured Capacity: 41083600896 (38.26 GB)
DFS Used: 40960 (40 KB)
Non DFS Used: 8360431616 (7.79 GB)
DFS Remaining: 32723128320(30.48 GB)
DFS Used%: 0%
DFS Remaining%: 79.65%
Last contact: Sat Dec 26 12:22:07 PST 2015

[email protected]:~/hadoop_1.2.1/hadoop-1.2.1/bin$

下面关于Hadoop的文章您也可能喜欢,不妨看看:

Ubuntu14.04下Hadoop2.4.1单机/伪分布式安装配置教程 http://www.codesec.net/Linux/2015-02/113487.htm

CentOS安装和配置Hadoop2.2.0 http://www.codesec.net/Linux/2014-01/94685.htm

Ubuntu 13.04上搭建Hadoop环境 http://www.codesec.net/Linux/2013-06/86106.htm

Ubuntu 12.10 +Hadoop 1.2.1版本集群配置 http://www.codesec.net/Linux/2013-09/90600.htm

Ubuntu上搭建Hadoop环境(单机模式+伪分布模式) http://www.codesec.net/Linux/2013-01/77681.htm

Ubuntu下Hadoop环境的配置 http://www.codesec.net/Linux/2012-11/74539.htm

单机版搭建Hadoop环境图文教程详解 http://www.codesec.net/Linux/2012-02/53927.htm

更多Hadoop相关信息见Hadoop 专题页面 http://www.codesec.net/topicnews.aspx?tid=13

本文地址:http://www.codesec.net/Linux/2016-09/135406.htm


Ubuntu 14.04 LTS下安装Hadoop 1.2.1(伪分布模式)

本文系统(linux)相关术语:linux系统 鸟哥的linux私房菜 linux命令大全 linux操作系统

分页:12
转载请注明
本文标题:Ubuntu 14.04 LTS下安装Hadoop 1.2.1(伪分布模式)
本站链接:http://www.codesec.net/view/481864.html
分享请点击:


1.凡CodeSecTeam转载的文章,均出自其它媒体或其他官网介绍,目的在于传递更多的信息,并不代表本站赞同其观点和其真实性负责;
2.转载的文章仅代表原创作者观点,与本站无关。其原创性以及文中陈述文字和内容未经本站证实,本站对该文以及其中全部或者部分内容、文字的真实性、完整性、及时性,不作出任何保证或承若;
3.如本站转载稿涉及版权等问题,请作者及时联系本站,我们会及时处理。
登录后可拥有收藏文章、关注作者等权限...
技术大类 技术大类 | 系统(linux) | 评论(0) | 阅读(101)