hadoop任务运行日志，hadoop任务日志

重要备注

处理代码

JobHistoryEventHandler#processDoneFiles(JobId)

日志在本地保留时间

yarn.nodemanager.delete.debug-delay-sec

本环境中参数：

${yarn.app.mapreduce.am.staging-dir} : /user

NodeManager local log包括即container log

示例：
-bash-4.1pwd/data/data11/yarn/logs/application1470221089121737080/container147022108912173708001000047−bash−4.1 ll -h
total 864K
-rw-r–r– 1 yarn yarn 0 Aug 26 10:08 stderr
-rw-r–r– 1 yarn yarn 0 Aug 26 10:08 stdout
-rw-r–r– 1 yarn yarn 864K Aug 26 15:19 syslog

JOB在运行的时候写在HDFS 的overview日志：

${yarn.app.mapreduce.am.staging-dir}/{username}/.staging/job_XXXXX_XXX/
示例：
-rw-r--r--  10 dsp supergroup   18397766 2016-08-26 10:07 /user/dsp/.staging/job_1470221089121_737080/job.jar
-rw-r--r--  10 dsp supergroup      22473 2016-08-26 10:07 /user/dsp/.staging/job_1470221089121_737080/job.split
-rw-r--r--   3 dsp supergroup        864 2016-08-26 10:07 /user/dsp/.staging/job_1470221089121_737080/job.splitmetainfo
-rw-r--r--   3 dsp supergroup     199357 2016-08-26 10:07 /user/dsp/.staging/job_1470221089121_737080/job.xml
-rw-r--r--   3 dsp supergroup        333 2016-08-26 10:07 /user/dsp/.staging/job_1470221089121_737080/job_1470221089121_737080.summary
-rw-r--r--   3 dsp supergroup      40623 2016-08-26 10:07 /user/dsp/.staging/job_1470221089121_737080/job_1470221089121_737080_1.jhist
-rw-r--r--   3 dsp supergroup     229223 2016-08-26 10:07 /user/dsp/.staging/job_1470221089121_737080/job_1470221089121_737080_1_conf.xml
drwx------   - dsp supergroup          0 2016-08-26 10:07 /user/dsp/.staging/job_1470221089121_737080/libjars

运行完成后，AM写入HDFS临时目录：

mapreduce.jobhistory.intermediate−done−dir/{username}目录，
其中mapreduce.jobhistory.intermediate-done-dir默认值是${yarn.app.mapreduce.am.staging-dir}/history/done_intermediate

示例：
-rwxrwx— 3 dsp hadoop 23629 2016-08-26 16:02 /user/history/done_intermediate/dsp/job_1470221089121_746317-1472198520907-dsp-from+NATIVE.HNATIVE_O…itle%2Cis_fund_dt_cust%28Stage-1472198529697-1-0-SUCCEEDED-root.dsp-1472198524249.jhist
-rwxrwx— 3 dsp hadoop 378 2016-08-26 16:02 /user/history/done_intermediate/dsp/job_1470221089121_746317.summary
-rwxrwx— 3 dsp hadoop 230331 2016-08-26 16:02 /user/history/done_intermediate/dsp/job_1470221089121_746317_conf.xml

日志收集线程进行日志整理写入的目录：

/user/history/done/2016/08/26/{xxx}
示例：
-rwxrwx— 3 dsp hadoop 23629 2016-08-26 16:02 /user/history/done/2016/08/26/000746/job_1470221089121_746317-1472198520907-dsp-from+NATIVE.HNATIVE_O…itle%2Cis_fund_dt_cust%28Stage-1472198529697-1-0-SUCCEEDED-root.dsp-1472198524249.jhist
-rwxrwx— 3 dsp hadoop 230331 2016-08-26 16:02 /user/history/done/2016/08/26/000746/job_1470221089121_746317_conf.xml

Hadoop 2.0中作业日志收集原理以及配置方法