hadoop，

hadoop笔记

王者归来-摩西。

下载

https://mirrors.tuna.tsinghua.edu.cn/apache/hadoop/common/stable/
hadoop-2.9.0.tar.gz

配置hadoop环境变量

HADOOP_HOME：D:\program\hadoop-2.9.0
path：%HADOOP_HOME%\bin

修改配置文件

core-site.xml

<configuration>
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/D:/program/hadoop-2.9.0/workplace/tmp</value>
    </property>
    <property>
        <name>dfs.name.dir</name>
        <value>/D:/program/hadoop-2.9.0/workplace/name</value>
    </property>
    <property>
        <name>fs.default.name</name>
        <value>hdfs://localhost:9000</value>
    </property>
</configuration>

mapred-site.xml

<configuration>
    <property>
       <name>mapreduce.framework.name</name>
       <value>yarn</value>
    </property>
    <property>
       <name>mapred.job.tracker</name>
       <value>hdfs://localhost:9001</value>
    </property>
</configuration>

hdfs-site.xml

<configuration>
    <!-- 这个参数设置为1，因为是单机版hadoop -->
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
    <property>
        <name>dfs.data.dir</name>
        <value>/D:/program/hadoop-2.9.0/workplace/data</value>
    </property>
</configuration>

yarn-site.xml

<configuration>
    <property>
       <name>yarn.nodemanager.aux-services</name>
       <value>mapreduce_shuffle</value>
    </property>
    <property>
       <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
       <value>org.apache.hadoop.mapred.ShuffleHandler</value>
    </property>
</configuration>

hadoop-env.cmd

@rem set JAVA_HOME=%JAVA_HOME%
set JAVA_HOME=C:\Progra~1\Java\jdk1.8.0_121

1、切换到etc/hadoop目录，运行hadoop-env.cmd
2、格式化HDFS文件系统，切换到bin目录然后执行命令：hdfs namenode -format
切换到 sbin目录 执行：start-dfs.cmd
查看hadoop管理页面：http://localhost:50070