Hive，MapReduce执行流程分析

Hive 任务提交流程

RM中事件转发器：AsyncDispatcher

通过rmcontext启动AM

this.rmContext.getDispatcher().getEventHandler()

.handle(new RMAppEvent(applicationId, RMAppEventType.START));

这里看到getDispatcher取rmDispatcher。

rmDispatcher初始化在ResourceManager类中：

rmDispatcher.register(SchedulerEventType.class, schedulerDispatcher);

// Register event handler for RmAppEvents
rmDispatcher.register(RMAppEventType.class,
new ApplicationEventDispatcher(rmContext));

// Register event handler for RmAppAttemptEvents
rmDispatcher.register(RMAppAttemptEventType.class,
new ApplicationAttemptEventDispatcher(rmContext));

// Register event handler for RmNodes
rmDispatcher.register(
RMNodeEventType.class, new NodeEventDispatcher(rmContext));

ApplicationEventDispatcher.handle –> RMAppImpl.handle 这里的start事件仅仅做了一个状态转换处理

YarnChild 启动

查看进程：

ps -ef|grep java|grep YarnChild

default_container_executor.sh 位置：

/data/yarn/local/usercache/hadoop/appcache/application_1486524626038_0307/container_e33_1486524626038_0307_01_000002/default_container_executor.sh

启动程序Java命令

$JAVA_HOME/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN  -agentlib:jdwp=transport=dt_socket,server=y,address=8002,suspend=y -Xmx1639m -Djava.io.tmpdir=$PWD/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/data/yarn/logs/application_1486524626038_0307/container_e33_1486524626038_0307_01_000003 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 10.20.9.42 41382 attempt_1486524626038_0307_m_000000_1 36283883716611 1>/data/yarn/logs/application_1486524626038_0307/container_e33_1486524626038_0307_01_000003/stdout 2>/data/yarn/logs/application_1486524626038_0307/container_e33_1486524626038_0307_01_000003/stderr

示例Map执行流程

Operator List:

MapOperator -> TableScanOperator -> FilterOperator -> SelectOperator -> LimitOperator -> ReduceSinkOperator -> OutputCollector out.collect(key,value)

MR 流程图