Hadoop实践 | 第一个例子WordCount，hadoopwordcount

Hadoop实践 | 第一个例子WordCount，hadoopwordcount

这是冲锋兵，下文紧跟另外两个例子

准备工作

导入eclipse maven ant hadoop-eclipse-plugin

mkdir ~/software
tar -zxvf /mnt/hgfs/share/eclipse-java-neon-3-linux-gtk-x86_64.tar.gz -C ~/software/

tar -zxvf /mnt/hgfs/share/apache-ant-1.10.1-bin.tar.gz -C ~/software/

tar -zxvf /mnt/hgfs/share/apache-maven-3.5.0-bin.tar.gz -C ~/software/

cp /mnt/hgfs/share/hadoop-eclipse-plugin-2.7.3.jar ~/software/eclipse/plugins/

配置环境

sudo vi /etc/profile

export  ECLIPSE_HOME=/home/jackherrick/software/eclipse
export PATH=.:$ECLIPSE_HOME:$PATH

export  ANT_HOME=/home/jackherrick/software/apache-ant-1.10.1
export PATH=.:$ANT_HOME/bin:$PATH

export  MAVEN_HOME=/home/jackherrick/software/apache-maven-3.5.0
export PATH=.:$MAVEN_HOMEE/bin:$PATH

终端可以直接输入 eclipse ，来运行起来

在eclipse中配置hadoop

window->preference->搜索hadoop map/reduce->添加安装目录

window->show view->other->搜索hadoop->添加Map/Reduce Location

如果第一次搞，DFS Location下是没有文件夹的，需要我们自己建立文件夹

把input文件夹传到HDFS上，HDFS（Hadoop Distributed File System），听名字也可以猜出来，它有着独立的文件系统。

下面是hadoop HDFS的常用文件操作命令,见下文。
我们这里用到的就是查看HDFS目录和把input文件夹放到HDFS

hadoop fs -ls  /
hdfs dfs -put ~/workspace/input  /

WordCount

好了，进入正题了，New一个Map/Reduce Project

package org.apache.hadoop.examples;  

import java.io.IOException;  
import java.util.Iterator;  
import java.util.StringTokenizer;  
import org.apache.hadoop.conf.Configuration;  
import org.apache.hadoop.fs.Path;  
import org.apache.hadoop.io.IntWritable;  
import org.apache.hadoop.io.Text;  
import org.apache.hadoop.mapreduce.Job;  
import org.apache.hadoop.mapreduce.Mapper;  
import org.apache.hadoop.mapreduce.Reducer;  
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;  
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;  
import org.apache.hadoop.util.GenericOptionsParser;  

public class WordCount {  
    public WordCount() {  
    }  

    public static void main(String[] args) throws Exception {  
        Configuration conf = new Configuration();  
        String[] otherArgs = (new GenericOptionsParser(conf, args)).getRemainingArgs();  
        if(otherArgs.length < 2) {  
            System.err.println("Usage: wordcount <in> [<in>...] <out>");  
            System.exit(2);  
        }  

        Job job = Job.getInstance(conf, "word count");  
        job.setJarByClass(WordCount.class);  
        job.setMapperClass(WordCount.TokenizerMapper.class);  
        job.setCombinerClass(WordCount.IntSumReducer.class);  
        job.setReducerClass(WordCount.IntSumReducer.class);  
        job.setOutputKeyClass(Text.class);  
        job.setOutputValueClass(IntWritable.class);  

        for(int i = 0; i < otherArgs.length - 1; ++i) {  
            FileInputFormat.addInputPath(job, new Path(otherArgs[i]));  
        }  

        FileOutputFormat.setOutputPath(job, new Path(otherArgs[otherArgs.length - 1]));  
        System.exit(job.waitForCompletion(true)?0:1);  
    }  

    public static class IntSumReducer extends Reducer<Text, IntWritable, Text, IntWritable> {  
        private IntWritable result = new IntWritable();  

        public IntSumReducer() {  
        }  

        public void reduce(Text key, Iterable<IntWritable> values, Reducer<Text, IntWritable, Text, IntWritable>.Context context) throws IOException, InterruptedException {  
            int sum = 0;  

            IntWritable val;  
            for(Iterator i$ = values.iterator(); i$.hasNext(); sum += val.get()) {  
                val = (IntWritable)i$.next();  
            }  

            this.result.set(sum);  
            context.write(key, this.result);  
        }  
    }  

    public static class TokenizerMapper extends Mapper<Object, Text, Text, IntWritable> {  
        private static final IntWritable one = new IntWritable(1);  
        private Text word = new Text();  

        public TokenizerMapper() {  
        }  

        public void map(Object key, Text value, Mapper<Object, Text, Text, IntWritable>.Context context) throws IOException, InterruptedException {  
            StringTokenizer itr = new StringTokenizer(value.toString());  

            while(itr.hasMoreTokens()) {  
                this.word.set(itr.nextToken());  
                context.write(this.word, one);  
            }  

        }  
    }  
}  

警告可以先不管

然后是导入配置文件

cp core-site.xml  ~/workspace/WordCount/src/
cp hdfs-site.xml  ~/workspace/WordCount/src/
cp log4j.properties  ~/workspace/WordCount/src/

在使用 Eclipse 运行 MapReduce 程序时，会读取 Hadoop-Eclipse-Plugin 的Advanced parameters作为 Hadoop 运行参数，如果未进行修改，则默认的参数其实就是单机（非分布式）参数，因此程序运行时是读取本地目录而不是HDFS 目录，就会提示Input 路径不存在。
所以需要将配置文件拷贝到项目中的 src 目录，来覆盖这些参数。让程序能够正确运行。log4j用于记录程序的输出日记，需要log4j.properties 这个配置文件，如果没有拷贝该文件到项目中，运行程序后在Console 面板中会出现警告提示：

右键点击刚创建的 WordCount.java，选择 Run As -> Run Configurations，在此处可以设置运行时的相关参数（如果 Java Application 下面没有 WordCount，那么需要先双击 Java Application）。切换到 “Arguments”栏，在Program arguments 处填写参数 “/input /output”。

也可以直接在代码中设置好输入参数。可将代码 main() 函数的 String[] otherArgs = new GenericOptionsParser(conf, args).getRemainingArgs(); 改为（没有测试）：

// String[] otherArgs = new GenericOptionsParser(conf, args).getRemainingArgs();  
String[] otherArgs=new String[]{"input","output"}; /* 直接设置输入参数 */