hadoop的集群搭建-摩杜云开发者社区

hadoop的集群搭建

所需要的tar包版本

hadoop-2.7.7.tar.gz
jdk-8u212-linux-x64.tar.gz

详情

hadoop完全分布式安装

更改主机名 (hostname) 主节点：master 两个从节点分别为(slave1和slave2)
更改 hosts

192.168.44.111 master
192.168.44.112 slave1
192.168.44.113 slave2

reboot 重启三个

免密登录：

ssh-keygen -t dsa -P "" （生成密钥三个都要）
将slave1和slave2的密钥发给master的authorized_keys 在将 authorized_keys 发给slave1和slave2
给予authorized_keys 权限

chmod 600 authorized_keys

解压 hadoop 和jdk 的tar包

tar -zxvf hadoop-2.7.7.tar.gz -C /opt/module
tar -zxvf jdk-8u212-linux-x64.tar.gz.gz -C /opt/module

配置环境变量

#java
export JAVA_HOME=/opt/module/jdk1.8.0_212
export JRE_HOME=/opt/module/jdk1.8.0_212/jre
export PATH=$PATH:$JAVA_HOME/bin
export CLASSPATH=.:$JAVA_HOME/dt.jar:$JAVA_HOME/tools.jar

#hadoop
export HADOOP_HOME=/opt/module/hadoop-2.7.7
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

source /etc/profile

将profile分发给slave1和slave2

source /etc/profile

创建三个文件夹

mkdir /opt/module/hadoop-2.7.7/tmp
mkdir -p /opt/module/hadoop-2.7.7/hdfs/name
mkdir -p /opt/module/hadoop-2.7.7/hdfs/data

cd /opt/module/hadoop-2.7.7/etc/hadoop(进行配置文件修改)

vi yarn-env.sh
vi core-site.xml

<property>
<name> fs.default.name </name>
<value>hdfs://master:9000</value>
</property>

<property>
<name>fs.defaultFS</name>
<value>hdfs://master:9000</value>
</property>

<property>
<name>hadoop.tmp.dir</name>
<value>/opt/module/hadoop-2.7.7/tmp</value>
</property>

vi hdfs-site.xml

<property>
        <name>dfs.namenode.name.dir</name>
        <value>file:/opt/module/hadoop-2.7.7/hdfs/name</value>
    </property>

    <property>
        <name>dfs.datanode.data.dir</name>
        <value>file:/opt/module/hadoop-2.7.7/hdfs/data</value>
    </property>

    <property>
        <name>dfs.replication</name>
        <value>1</value>
 </property>

cp mapred-site.xml.template mapred-site.xml 将mapred-site.xml.template文件复制到当前目录，并重命名为mapred-site.xml
vi mapred-site.xml

<property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
</property>

vi yarn-site.xml

<property>
        <name>yarn.resourcemanager.hostname</name>
        <value>master</value>
</property>

<property>
		<name>yarn.nodemanager.aux-services</name>
		<value>mapreduce_shuffle</value>
</property>

vi slaves

slave1

slave2

7. 将hadoop整体分发给slave1和slave2