学习笔记:进步一认识mesos(部分翻译)
2016-07-24 09:52
447 查看
Source: Mesos in action.
我就喜欢图多的书!!!
工作流程详解:
他的实际工作流程:
1. The Mesos slave offers itsavailable CPU, memory, and disk to the Mesos master in the form of a resourceoffer.
2. The Mesos master’s allocationmodule—or scheduling algorithm—decides which frameworks—or applications—tooffer the resources to.
3. In this particular case, theSpark scheduler doesn’t have any jobs to run on the cluster. It rejects theresource offer, allowing the master to offer the resources to another frameworkthat might have some work to do.
4. Now consider a usersubmitting a Spark job to be run on the cluster. The scheduler accepts the joband waits for a resource offer that satisfies the workload.
5. The Spark scheduler accepts aresource offer from the Mesos master, and launches one or more tasks on anavailable Mesos slave. These tasks are launched within a container, providing isolation between thevarious tasks that might be running on a given Mesos
slave.
Resource offer方面:
Slave会周期的向Master发送resource offer。
//那么多offer怎么协调,这也是问题啊。看源码master的这个部分。
Scheduling方面:
对应用分配资源的方法:Dominant Resource Fairness(DRF)
In a nutshell, DRFseeks to maximize the minimum dominant share across all users. For example, ifuser A runs CPU-heavy tasks and user B runs memory-heavy tasks, DRF attempts toequalize user A’s share of CPUs with user B’s share of memory. In thesingle-resource
case, DRF reduces to max-min fairness for that resource.
方法论文来源:www.usenix.org/legacy/events/nsdi11/tech/full_papers/Ghodsi.pdf.
Resource Isolation方面:
采用linux中的cgroup或者docker container的方法。Container技术可以理解成在一个物理机上同时运行多个虚拟机,但没有开销或者真的在运行整个操作系统。
NOTE:为其他的POSIX-COMPLIANT系统提供posix/cpu, posix/mem, posix/disk的方法。但实际上没有隔离作用。倒是可以用来监测资源。
与传统方法上的不同点:
Docker技术
相关资料:
Linux control groups:
www.kernel.org/doc/documentation/cgroup-v1/cgroups.txt
Docker: https://docs.docker.com
Mesos Architecture
如果有两个类型的应用Mesos是怎么运行的呢
在Mesos上运行一个Spark的事件流图
资源分配部分:
Mesos资源分配的时候允许给slave自己有保留资源并且会给他一个可是别的role
--roles="dev,stage,prod"
除了role这个标签(先这么理解),还有一个weight.
--weights="dev=10,stage=20,prod=30"
还有一个标签是来说明现在还有多少资源可用的
--resources="cpus(prod):8;mem(prod):16384; disk(prod):65536"
还可以定制资源。
我就喜欢图多的书!!!
工作流程详解:
他的实际工作流程:
1. The Mesos slave offers itsavailable CPU, memory, and disk to the Mesos master in the form of a resourceoffer.
2. The Mesos master’s allocationmodule—or scheduling algorithm—decides which frameworks—or applications—tooffer the resources to.
3. In this particular case, theSpark scheduler doesn’t have any jobs to run on the cluster. It rejects theresource offer, allowing the master to offer the resources to another frameworkthat might have some work to do.
4. Now consider a usersubmitting a Spark job to be run on the cluster. The scheduler accepts the joband waits for a resource offer that satisfies the workload.
5. The Spark scheduler accepts aresource offer from the Mesos master, and launches one or more tasks on anavailable Mesos slave. These tasks are launched within a container, providing isolation between thevarious tasks that might be running on a given Mesos
slave.
Resource offer方面:
Slave会周期的向Master发送resource offer。
//那么多offer怎么协调,这也是问题啊。看源码master的这个部分。
Scheduling方面:
对应用分配资源的方法:Dominant Resource Fairness(DRF)
In a nutshell, DRFseeks to maximize the minimum dominant share across all users. For example, ifuser A runs CPU-heavy tasks and user B runs memory-heavy tasks, DRF attempts toequalize user A’s share of CPUs with user B’s share of memory. In thesingle-resource
case, DRF reduces to max-min fairness for that resource.
方法论文来源:www.usenix.org/legacy/events/nsdi11/tech/full_papers/Ghodsi.pdf.
Resource Isolation方面:
采用linux中的cgroup或者docker container的方法。Container技术可以理解成在一个物理机上同时运行多个虚拟机,但没有开销或者真的在运行整个操作系统。
NOTE:为其他的POSIX-COMPLIANT系统提供posix/cpu, posix/mem, posix/disk的方法。但实际上没有隔离作用。倒是可以用来监测资源。
与传统方法上的不同点:
Docker技术
相关资料:
Linux control groups:
www.kernel.org/doc/documentation/cgroup-v1/cgroups.txt
Docker: https://docs.docker.com
Mesos Architecture
如果有两个类型的应用Mesos是怎么运行的呢
在Mesos上运行一个Spark的事件流图
资源分配部分:
Mesos资源分配的时候允许给slave自己有保留资源并且会给他一个可是别的role
--roles="dev,stage,prod"
除了role这个标签(先这么理解),还有一个weight.
--weights="dev=10,stage=20,prod=30"
还有一个标签是来说明现在还有多少资源可用的
--resources="cpus(prod):8;mem(prod):16384; disk(prod):65536"
还可以定制资源。
相关文章推荐
- Apache Isis 1.4.0 发布,领域驱动开发框架
- RH436 Day3 课后总结
- Linux快速构建apache web服务器
- Awstats处理多apache日志
- 安装perl模块小窍门
- Apache静态编译与动态编译的区别
- PHP+Apache在Windows 9x下的安装和配置
- Apache服务器配置全攻略
- Apache Web让JSP“动”起来
- Linux Apache+MySQL+PHP
- 建立Apache+PHP+MySQL数据库驱动的动态网站
- apache 环境下 php 的配置注意事项
- 在RedHat下安装apache jserv 1.1.2方法
- windows8.1下Apache+Php+MySQL配置步骤
- windows中PHP5.2.14以及apache2.2.16安装配置方法第1/2页
- 在Windows XP下安装Apache+MySQL+PHP环境
- 解析阿里云ubuntu12.04环境下配置Apache+PHP+PHPmyadmin+MYsql
- apache rewrite防盗链三例
- Apache 配置详解(最好的APACHE配置教程)
- Apache+php+mysql在windows下的安装与配置(图文)第1/2页