hadoop，

和通数据库htsjk.Com2019-11-04 21:57 来源:未知阅读:18985 评论 345 热度3

标签：

hadoop，

<div id="article_content" class="article_content clearfix csdn-tracking-statistics" data-pid="blog" data-mod="popu_307" data-dsm="post">
<div class="markdown_views">
<blockquote>
介绍如何在Intellij Idea中通过创建maven工程配置MapReduce的编程环境。
</blockquote>

我使用的软件版本如下:

<ol>
<li>Intellij Idea 2017.1</li>
<li>Maven 3.3.9</li>
<li>Hadoop伪分布式环境( 安装教程可参考<a href="http://blog.csdn.net/napoay/article/details/54136398" rel="nofollow" target="_blank">这里</a>)</li>
</ol>

<h1 id="二创建maven工程"><a name="t1"></a>二、创建maven工程</h1>

打开Idea,file->new->Project,左侧面板选择maven工程。(如果只跑MapReduce创建java工程即可，不用勾选Creat from archetype，如果想创建web工程或者使用骨架可以勾选) 
<img src="https://img-blog.csdn.net/20170330195641898?watermark/2/text/aHR0cDovL2Jsb2cuY3Nkbi5uZXQv/font/5a6L5L2T/fontsize/400/fill/I0JBQkFCMA==/dissolve/70/gravity/SouthEast" alt="这里写图片描述" title=""> 
设置GroupId和ArtifactId，下一步。 
<img src="https://img-blog.csdn.net/20170330195925542?watermark/2/text/aHR0cDovL2Jsb2cuY3Nkbi5uZXQvbmFwb2F5/font/5a6L5L2T/fontsize/400/fill/I0JBQkFCMA==/dissolve/70/gravity/SouthEast" alt="这里写图片描述" title=""> 
设置工程存储路径，下一步。 
<img src="https://img-blog.csdn.net/20170330200053760?watermark/2/text/aHR0cDovL2Jsb2cuY3Nkbi5uZXQvbmFwb2F5/font/5a6L5L2T/fontsize/400/fill/I0JBQkFCMA==/dissolve/70/gravity/SouthEast" alt="这里写图片描述" title=""> 
Finish之后，空白工程的路径如下图所示。

完整的工程路径如下图所示： 
<img src="https://img-blog.csdn.net/20170330204504371?watermark/2/text/aHR0cDovL2Jsb2cuY3Nkbi5uZXQvbmFwb2F5/font/5a6L5L2T/fontsize/400/fill/I0JBQkFCMA==/dissolve/70/gravity/SouthEast" alt="这里写图片描述" title="">

<h1 id="三添加maven依赖"><a name="t2"></a>三、添加maven依赖</h1>

在pom.xml添加依赖，对于hadoop 2.7.3版本的hadoop，需要的jar包有以下几个：

<ul>
<li>hadoop-common</li>
<li>hadoop-hdfs</li>
<li>hadoop-mapreduce-client-core</li>
<li>hadoop-mapreduce-client-jobclient</li>
<li>log4j( 打印日志)

pom.xml中的依赖如下:</li>
</ul>

<pre class="prettyprint" name="code"><code class="hljs xml has-numbering"> <dependencies>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.12</version>
<scope>test</scope>
</dependency>

<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>2.7.3</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-hdfs</artifactId>
<version>2.7.3</version>
</dependency>

<h1 id="四配置log4j"><a name="t3"></a>四、配置log4j</h1>

在<code>src/main/resources</code>目录下新增log4j的配置文件<code>log4j.properties</code>，内容如下：

<pre class="prettyprint" name="code"><code class="hljs avrasm has-numbering">log4j.rootLogger = debug,stdout

### 输出信息到控制抬 ###
log4j.appender.stdout = org.apache.log4j.ConsoleAppender
log4j.appender.stdout.Target = System.out
log4j.appender.stdout.layout = org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern = [%-5p] %d{yyyy-MM-dd HH:mm:ss,SSS} method:%l%n%m%n
</code><ul class="pre-numbering"><li >1</li><li >2</li><li >3</li><li >4</li><li >5</li><li >6</li><li >7</li><li >8</li></ul></pre>

<h1 id="五启动hadoop"><a name="t4"></a>五、启动Hadoop</h1>

启动Hadoop，运行命令：

<pre class="prettyprint" name="code"><code class="hljs sql has-numbering">cd hadoop-2.7.3/
./sbin/start-all.sh</code><ul class="pre-numbering"><li >1</li><li >2</li></ul></pre>

访问<a href="http://localhost:50070" rel="nofollow" target="_blank">http://localhost:50070/</a>查看hadoop是否正常启动。

<h1 id="六运行wordcount从本地读取文件"><a name="t5"></a>六、运行WordCount(从本地读取文件)</h1>

在工程根目录下新建input文件夹，input文件夹下新增dream.txt，随便写入一些单词:

<pre class="prettyprint" name="code"><code class="hljs livecodeserver has-numbering">I have a dream
a dream</code><ul class="pre-numbering"><li >1</li><li >2</li></ul></pre>

在src/main/java目录下新建包，新增FileUtil.java，创建一个删除output文件的函数，以后就不用手动删除了。内容如下:

<pre class="prettyprint" name="code"><code class="hljs java has-numbering">package com.mrtest.hadoop;

import java.io.File;

/**
* Created by bee on 3/25/17.
*/
public class FileUtil {

public static boolean deleteDir(String path) {
File dir = new File(path);
if (dir.exists()) {
for (File f : dir.listFiles()) {
if (f.isDirectory()) {
deleteDir(f.getName());
} else {
f.delete();
}
}
dir.delete();
return true;
} else {
System.out.println("文件(夹)不存在!");
return false;
}
}

}</code><ul class="pre-numbering"><li >1</li><li >2</li><li >3</li><li >4</li><li >5</li><li >6</li><li >7</li><li >8</li><li >9</li><li >10</li><li >11</li><li >12</li><li >13</li><li >14</li><li >15</li><li >16</li><li >17</li><li >18</li><li >19</li><li >20</li><li >21</li><li >22</li><li >23</li><li >24</li><li >25</li><li >26</li><li >27</li><li >28</li></ul></pre>

编写WordCount的MapReduce程序WordCount.java，内容如下:

<pre class="prettyprint" name="code"><code class="hljs avrasm has-numbering">package com.mrtest.hadoop;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;

import java.io.IOException;
import java.util.Iterator;
import java.util.StringTokenizer;

/**
* Created by bee on 3/25/17.
*/
public class WordCount {

public static class TokenizerMapper extends
Mapper<Object, Text, Text, IntWritable> {

public static final IntWritable one = new IntWritable(1);
private Text word = new Text();

public void map(Object key, Text value, Context context)
throws IOException, InterruptedException {
StringTokenizer itr = new StringTokenizer(value.toString());
while (itr.hasMoreTokens()) {
this.word.set(itr.nextToken());
context.write(this.word, one);
}
}

}

public static class IntSumReduce extends
Reducer<Text, IntWritable, Text, IntWritable> {
private IntWritable result = new IntWritable();

public void reduce(Text key, Iterable<IntWritable> values,
Context context)
throws IOException, InterruptedException {
int sum = 0;
IntWritable val;
for (Iterator i = values.iterator(); i.hasNext(); sum += val.get()) {
val = (IntWritable) i.next();
}
this.result.set(sum);
context.write(key, this.result);
}
}

public static void main(String[] args)
throws IOException, ClassNotFoundException, InterruptedException {

FileUtil.deleteDir("output");
Configuration conf = new Configuration();

String[] otherArgs = new String[]{"input/dream.txt","output"};
if (otherArgs.length != 2) {
System.err.println("Usage:Merge and duplicate removal <in> <out>");
System.exit(2);
}

Job job = Job.getInstance(conf, "WordCount");
job.setJarByClass(WordCount.class);
job.setMapperClass(WordCount.TokenizerMapper.class);
job.setReducerClass(WordCount.IntSumReduce.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
FileInputFormat.addInputPath(job, new Path(otherArgs[0]));
FileOutputFormat.setOutputPath(job, new Path(otherArgs[1]));
System.exit(job.waitForCompletion(true) ? 0 : 1);
}
}
</code><ul class="pre-numbering"><li >1</li><li >2</li><li >3</li><li >4</li><li >5</li><li >6</li><li >7</li><li >8</li><li >9</li><li >10</li><li >11</li><li >12</li><li >13</li><li >14</li><li >15</li><li >16</li><li >17</li><li >18</li><li >19</li><li >20</li><li >21</li><li >22</li><li >23</li><li >24</li><li >25</li><li >26</li><li >27</li><li >28</li><li >29</li><li >30</li><li >31</li><li >32</li><li >33</li><li >34</li><li >35</li><li >36</li><li >37</li><li >38</li><li >39</li><li >40</li><li >41</li><li >42</li><li >43</li><li >44</li><li >45</li><li >46</li><li >47</li><li >48</li><li >49</li><li >50</li><li >51</li><li >52</li><li >53</li><li >54</li><li >55</li><li >56</li><li >57</li><li >58</li><li >59</li><li >60</li><li >61</li><li >62</li><li >63</li><li >64</li><li >65</li><li >66</li><li >67</li><li >68</li><li >69</li><li >70</li><li >71</li><li >72</li><li >73</li><li >74</li><li >75</li><li >76</li><li >77</li><li >78</li><li >79</li><li >80</li><li >81</li></ul></pre>

运行完毕以后，会在工程根目录下增加一个output文件夹，打开output/part-r-00000，内容如下:

<pre class="prettyprint" name="code"><code class="hljs mathematica has-numbering">I 1
a 2
dream 2
have 1</code><ul class="pre-numbering"><li >1</li><li >2</li><li >3</li><li >4</li></ul></pre>

这里在main函数中新增了一个String类型的数组，如果想用main函数的args数组接受参数，在运行时指定输入和输出路径也是可以的。运行WordCount之前，配置Configuration并指定Program arguments即可。 
<img src="https://img-blog.csdn.net/20170330203027273?watermark/2/text/aHR0cDovL2Jsb2cuY3Nkbi5uZXQvbmFwb2F5/font/5a6L5L2T/fontsize/400/fill/I0JBQkFCMA==/dissolve/70/gravity/SouthEast" alt="这里写图片描述" title="">

<hr>

<h1 id="七运行wordcount从hdfs读取文件"><a name="t6"></a>七、运行WordCount(从HDFS读取文件)</h1>

在HDFS上新建文件夹:

<pre class="prettyprint" name="code"><code class="hljs perl has-numbering">hadoop fs -mkdir /worddir</code><ul class="pre-numbering"><li >1</li></ul></pre>

如果出现Namenode安全模式导致的不能创建文件夹提示：

<pre class="prettyprint" name="code"><code class="hljs sql has-numbering">mkdir: Cannot create directory /worddir. Name node is in safe mode.</code><ul class="pre-numbering"><li >1</li></ul></pre>

运行以下命令关闭safe mode:

<pre class="prettyprint" name="code"><code class="hljs lasso has-numbering">hadoop dfsadmin -safemode leave</code><ul class="pre-numbering"><li >1</li></ul></pre>

上传本地文件:

<pre class="prettyprint" name="code"><code class="hljs livecodeserver has-numbering">hadoop fs -put dream.txt /worddir</code><ul class="pre-numbering"><li >1</li></ul></pre>

修改otherArgs参数，指定输入为文件在HDFS上的路径:

<pre class="prettyprint" name="code"><code class="hljs javascript has-numbering">String[] otherArgs = new String[]{"hdfs://localhost:9000/worddir/dream.txt","output"};</code><ul class="pre-numbering"><li >1</li></ul></pre>

代码下载地址:<a href="http://download.csdn.net/detail/napoay/9799523" rel="nofollow" target="_blank">http://download.csdn.net/detail/napoay/9799523</a> </div>
<link rel="stylesheet" href="https://csdnimg.cn/release/phoenix/template/css/markdown_views-ea0013b516.css">
</div>