Hadoop MapReduce Next Generation - Writing YARN Applications
2014-02-08 15:32
429 查看
The general concept is that an 'Application Submission Client' submits an 'Application' to the YARN Resource Manager. The client communicates with the ResourceManager using the 'ApplicationClientProtocol'
to first acquire a new 'ApplicationId' if needed via ApplicationClientProtocol#getNewApplication and then submit the 'Application' to be run via ApplicationClientProtocol#submitApplication. As part of the ApplicationClientProtocol#submitApplication call, the
client needs to provide sufficient information to the ResourceManager to 'launch' the application's first container i.e. the ApplicationMaster. You need to provide information such as the details about the local files/jars that need to be available for your
application to run, the actual command that needs to be executed (with the necessary command line arguments), any Unix environment settings (optional), etc. Effectively, you need to describe the Unix process(es) that needs to be launched for your ApplicationMaster.
The YARN ResourceManager will then launch the ApplicationMaster (as specified) on an allocated container. The ApplicationMaster is then expected to communicate with the ResourceManager
using the 'ApplicationMasterProtocol'. Firstly, the ApplicationMaster needs to register itself with the ResourceManager. To complete the task assigned to it, the ApplicationMaster can then request for and receive containers via ApplicationMasterProtocol#allocate.
After a container is allocated to it, the ApplicationMaster communicates with the NodeManager using ContainerManager#startContainer to launch the container for its task. As part of launching this container, the ApplicationMaster has to specify the ContainerLaunchContext
which, similar to the ApplicationSubmissionContext, has the launch information such as command line specification, environment, etc. Once the task is completed, the ApplicationMaster has to signal the ResourceManager of its completion via the ApplicationMasterProtocol#finishApplicationMaster.
Meanwhile, the client can monitor the application's status by querying the ResourceManager or by directly querying the ApplicationMaster if it supports such a service. If needed, it
can also kill the application via ApplicationClientProtocol#forceKillApplication.
Ref: http://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/WritingYarnApplications.html
to first acquire a new 'ApplicationId' if needed via ApplicationClientProtocol#getNewApplication and then submit the 'Application' to be run via ApplicationClientProtocol#submitApplication. As part of the ApplicationClientProtocol#submitApplication call, the
client needs to provide sufficient information to the ResourceManager to 'launch' the application's first container i.e. the ApplicationMaster. You need to provide information such as the details about the local files/jars that need to be available for your
application to run, the actual command that needs to be executed (with the necessary command line arguments), any Unix environment settings (optional), etc. Effectively, you need to describe the Unix process(es) that needs to be launched for your ApplicationMaster.
The YARN ResourceManager will then launch the ApplicationMaster (as specified) on an allocated container. The ApplicationMaster is then expected to communicate with the ResourceManager
using the 'ApplicationMasterProtocol'. Firstly, the ApplicationMaster needs to register itself with the ResourceManager. To complete the task assigned to it, the ApplicationMaster can then request for and receive containers via ApplicationMasterProtocol#allocate.
After a container is allocated to it, the ApplicationMaster communicates with the NodeManager using ContainerManager#startContainer to launch the container for its task. As part of launching this container, the ApplicationMaster has to specify the ContainerLaunchContext
which, similar to the ApplicationSubmissionContext, has the launch information such as command line specification, environment, etc. Once the task is completed, the ApplicationMaster has to signal the ResourceManager of its completion via the ApplicationMasterProtocol#finishApplicationMaster.
Meanwhile, the client can monitor the application's status by querying the ResourceManager or by directly querying the ApplicationMaster if it supports such a service. If needed, it
can also kill the application via ApplicationClientProtocol#forceKillApplication.
Ref: http://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/WritingYarnApplications.html
相关文章推荐
- hadoop 0.23 YARN分布式程序的编写 (Hadoop MapReduce Next Generation - Writing YARN Applications)
- hadoop 0.23 YARN分布式程序的编写 (Hadoop MapReduce Next Generation - Writing YARN Applications)
- 云计算(二十五)- Hadoop MapReduce Next Generation - Writing YARN Applications
- Hadoop 新 MapReduce 框架 Yarn 详解
- hadoop2提交到Yarn: Mapreduce执行过程分析2
- hadoop备战:yarn框架的简介(mapreduce2)
- Hadoop-2.2.0中文文档——MapReduce--写YARN应用
- Hadoop 新 MapReduce 框架 Yarn 详解
- Hadoop环境搭建三之配置部署启动YARN以及在YARN上运行MapReduce程序
- hadoop2提交到Yarn: Mapreduce执行过程分析1
- Hadoop 新 MapReduce 框架 Yarn 详解
- Hadoop 新 MapReduce 框架 Yarn 详解
- hadoop cdh4.5 mapreduce yarn 配置文件 (接上文)
- Hadoop 新 MapReduce 框架 Yarn 详解
- Hadoop 新 MapReduce 框架 Yarn 详解
- hadoop2提交到Yarn: Mapreduce执行过程reduce分析3
- Hadoop 新 MapReduce 框架 Yarn 详解
- Hadoop 新 MapReduce 框架 Yarn 详解
- hadoop2提交到Yarn: Mapreduce执行过程分析
- hadoop2提交到Yarn: Mapreduce执行过程分析2