Ubuntu20.04本地配置Hadoop环境

Author Avatar
MorphLing 10月 21, 2021


环境配置

  • Ubuntu20.04
  • jdk版本：jdk1.8.0_311

https://www.oracle.com/java/technologies/downloads/#java8

  • hadoop版本：hadoop-2.3.2

http://archive.apache.org/dist/hadoop/common/hadoop-2.8.3/hadoop-2.8.3.tar.gz

hadoop配置

注意ip设置为本地ip

hadoop/etc/hadoop/hdfs.site.xml

<configuration>
    <property>
        <name>dfs.replication</name>
        <value>3</value>
    </property>
</configuration>

…/yarn-site.xml

<configuration>
    <property>
        <name>yarn.resourcemanager.hostname</name>
        <value>192.168.130.133</value>
    </property>
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
</configuration>

…/hadoop-env.sh

配置JAVA_HOME路径

export JAVA_HOME=/home/lyf/Downloads/jdk1.8.0_311

…/mapred-site.xml（手动创建）

<configuration>
    <property>
        <name> mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
</configuration>

…/core-site.xml

<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://192.168.130.133</value>
    </property>
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/home/lyf/Downloads/hadoop/tmp</value>
    </property>
</configuration>

启动hadoop服务

配置环境变量~/.bashrc

export PATH="/home/lyf/Downloads/hadoop/bin:$PATH"
export PATH="/home/lyf/Downloads/hadoop/sbin:$PATH"

创建tmp文件夹

cd /hadoop
mkdir tmp

启动hadoop服务器

hdfs namenode -format
start-all.sh

关闭hadoop服务器

stop-all.sh

datanode起不来的解决方法：

> cd /hadoop/tmp
> rm -rf /dfs
> hdfs namenode -format
> start-all.sh

验证启动成功与否

image-20211021203425174

访问localhost:8088 ResourceManager

image-20211021203453842