OpenStack使用Bosh部署CloudFoundry(四)—使用Micro Bosh部署Bosh

版本说明：

CloudFoundry：V2版本-v149+（本文后续章节中的Manifest适用于149+版本的CF Release）



OpenStack：Grizzly版本+nova-network

CloudFoundry部署交流QQ群：176302388

本文介绍如何使用Micro Bosh来部署Bosh，部署需要创建8个虚拟机来部署8个Bosh组件，所以，至少得准备好8个Fixed IP，后边使用Bosh的时候需要访问Director组件及配置DNS，所以准备至少2个Floating IP，本文中为（10.68.19.131和10.68.19.132），Bosh组件的详细介绍可以参看下边连接。

BOSH组件介绍：http://docs.cloudfoundry.com/docs/running/bosh/components/

下面是部署步骤：

1、上传Bosh Stemcell到Micro Bosh中

root@bosh-cli:~/bosh-workspace/stemcells# bosh upload stemcell bosh-stemcell-latest-openstack-kvm-ubuntu.tgz

Verifying stemcell...
File exists and readable                                     OK
Using cached manifest...
Stemcell properties                                          OK

Stemcell info
-------------
Name:    bosh-openstack-kvm-ubuntu
Version: 1478

Checking if stemcell already exists...
No

Uploading stemcell...

latest-bosh-s: 100% |ooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo| 251.3MB  11.3MB/s Time: 00:00:22

Director task 1

Update stemcell
extracting stemcell archive (00:00:06)
verifying stemcell manifest (00:00:00)
checking if this stemcell already exists (00:00:00)
uploading stemcell bosh-stemcell/776 to the cloud (00:01:53)
save stemcell bosh-stemcell/776 (f207bfcd-f0cb-4ea2-b753-022e6234ebfa) (00:00:00)
Done                    5/5 00:01:59

Task 1 done
Started         2013-12-16 09:51:59 UTC
Finished        2013-12-16 09:53:58 UTC
Duration        00:01:59

Stemcell uploaded and created

2、下载Bosh Release代码并打包

root@bosh-cli:~# cd ~/bosh-workspace
root@bosh-cli:~# git clone git://github.com/cloudfoundry/bosh.git

耗费时间较长，请耐心等待。

root@bosh-cli:~# cd ~/bosh-workspace/bosh
root@bosh-cli:~# apt-get install libxslt-dev libxml2-dev libmysql-ruby libmysqlclient-dev libpq-dev
root@bosh-cli:~# bundle install --local
root@bosh-cli:~# bundle exec rake release:create_dev_release

可以先修改~/bosh-workspace/bosh/Gemfile的source为淘宝源：

source 'http://ruby.taobao.org'

打包结束得到以下输出：

bosh create release --force
Syncing blobs...

Building DEV release
---------------------------------

....

Release version: 13.1-dev
Release manifest: /root/bosh-workspace/bosh/release/dev_releases/bosh-13.1-dev.yml

3、上传Bosh Release包到Micro Bosh中

root@bosh-cli:~/bosh-workspace# bosh upload release ~/bosh-workspace/bosh/release/dev_releases/bosh-13.1-dev.yml

成功后得到以下输出：

....

Creating new jobs
blobstore/3.1-dev (00:00:00)
powerdns/3.1-dev (00:00:00)
redis/3 (00:00:00)
nats/5 (00:00:00)
postgres/4.1-dev (00:00:00)
director/11.2-dev (00:00:00)
health_monitor/5.1-dev (00:00:00)
registry/0.1-dev (00:00:00)
Done                    8/8 00:00:00

Release has been created
bosh/13.1-dev (00:00:00)
Done                    1/1 00:00:00

Task 2 done
Started         2013-12-16 10:06:26 UTC
Finished        2013-12-16 10:06:46 UTC
Duration        00:00:20

Release uploaded

4、确认已经上传的stemcell和release

root@bosh-cli:~# bosh stemcells

+---------------------------+---------+--------------------------------------+
| Name                      | Version | CID                                  |
+---------------------------+---------+--------------------------------------+
| bosh-openstack-kvm-ubuntu | 1478    | e64c6da5-f2ab-48f2-b03d-79c29d1acc43 |
+---------------------------+---------+--------------------------------------+

Stemcells total: 1
root@bosh-cli:~# bosh releases

+------+-----------+-------------+
| Name | Versions  | Commit Hash |
+------+-----------+-------------+
| bosh | 13.1-dev* | a8e43f41+   |
+------+-----------+-------------+
(*) Currently deployed
(+) Uncommitted changes

Releases total: 1

5、创建部署Bosh的Manifest文件

创建目录

root@bosh-cli:~# mkdir -p ~/bosh-workspace/deployments/bosh-openstack
root@bosh-cli:~# cd ~/bosh-workspace/deployments/bosh-openstack

复制模板文件

root@bosh-cli:~# cp ~/bosh-workspace/bosh/release/examples/bosh-openstack-dynamic.yml bosh-openstack.yml

如果OpenStack中采用的是quantum网络组件，则使用下边的模板文件

root@bosh-cli:~# cp ~/bosh-workspace/bosh/release/examples/bosh-openstack-manual.yml bosh-openstack.yml

本次安装中使用nova-network网络组件，所以采用bosh-openstack-dynamic.yml，修改其中标记有# CHANGE 的配置项，下边是本环境中的配置，大家可参考：

---
name: bosh-openstack
director_uuid: 5d8c682c-290a-4194-bfc6-60248567ceab # CHANGE

release:
name: bosh
version: latest

compilation:
workers: 3
network: default
reuse_compilation_vms: true
cloud_properties:
instance_type: v1.compile # CHANGE

update:
canaries: 1
canary_watch_time: 3000-120000
update_watch_time: 3000-120000
max_in_flight: 4
max_errors: 1

networks:
- name: floating
type: vip
cloud_properties: {}
- name: default
type: dynamic
cloud_properties: {}

resource_pools:
- name: common
network: default
size: 8
stemcell:
name: bosh-openstack-kvm-ubuntu
version: latest
cloud_properties:
instance_type: v1.micro # CHANGE

jobs:
- name: nats
template: nats
instances: 1
resource_pool: common
networks:
- name: default
default: [dns, gateway]

- name: redis
template: redis
instances: 1
resource_pool: common
networks:
- name: default
default: [dns, gateway]

- name: postgres
template: postgres
instances: 1
resource_pool: common
persistent_disk: 16384
networks:
- name: default
default: [dns, gateway]

- name: powerdns
template: powerdns
instances: 1
resource_pool: common
networks:
- name: default
default: [dns, gateway]
- name: floating
static_ips:
- 10.68.19.131 # CHANGE

- name: blobstore
template: blobstore
instances: 1
resource_pool: common
networks:
- name: default
default: [dns, gateway]

- name: director
template: director
instances: 1
resource_pool: common
persistent_disk: 16384
networks:
- name: default
default: [dns, gateway]
- name: floating
static_ips:
- 10.68.19.132 # CHANGE

- name: registry
template: registry
instances: 1
resource_pool: common
networks:
- name: default
default: [dns, gateway]

- name: health_monitor
template: health_monitor
instances: 1
resource_pool: common
networks:
- name: default
default: [dns, gateway]

properties:
nats:
address: 0.nats.default.bosh-openstack.microbosh
user: nats
password: nats

redis:
address: 0.redis.default.bosh-openstack.microbosh
password: redis

postgres: &bosh_db
host: 0.postgres.default.bosh-openstack.microbosh
user: postgres
password: postgres
database: bosh

dns:
address: 10.68.19.131 # CHANGE
db: *bosh_db
recursor: 10.68.19.130 # CHANGE

blobstore:
address: 0.blobstore.default.bosh-openstack.microbosh
agent:
user: agent
password: agent
director:
user: director
password: director

director:
name: bosh
address: 0.director.default.bosh-openstack.microbosh
db: *bosh_db

registry:
address: 0.registry.default.bosh-openstack.microbosh
db: *bosh_db
http:
user: registry
password: registry

hm:
http:
user: hm
password: hm
director_account:
user: admin
password: admin
resurrector_enabled: true

ntp:
- 0.north-america.pool.ntp.org
- 1.north-america.pool.ntp.org

openstack:
auth_url: http://10.68.19.61:5000/v2.0 # CHANGE
username: VF # CHANGE
api_key: 1 # CHANGE
tenant: VF_CF # CHANGE
region: myregion # CHANGE
default_security_groups: ["default"] # CHANGE
default_key_name: vkey # CHANGE

注意：director_uuid是Micro Bosh的Director的UUID，可以使用bosh status查看并复制到Manifest文件中。

6、部署Bosh

设置Bosh的部署文件

root@bosh-cli:~/bosh-workspace/deployments# bosh deployment ~/bosh-workspace/deployments/bosh-openstack/bosh-openstack.yml
Deployment set to `/root/bosh-workspace/deployments/bosh-openstack/bosh-openstack.yml'

执行部署命令

root@bosh-cli:~/bosh-workspace/deployments# bosh deploy

部署时可能会遇到错误RateLimit报错，报错内容如下：

E, [2013-12-15T06:13:32.599434 #6477] [task:3] ERROR -- : OpenStack API Request Entity Too Large error. Check task debug log for details.
/var/vcap/packages/director/gem_home/gems/bosh_openstack_cpi-1.5.0.pre.1478/lib/cloud/openstack/helpers.rb:20:in `cloud_error'
/var/vcap/packages/director/gem_home/gems/bosh_openstack_cpi-1.5.0.pre.1478/lib/cloud/openstack/helpers.rb:39:in `rescue in with_openstack'
/var/vcap/packages/director/gem_home/gems/bosh_openstack_cpi-1.5.0.pre.1478/lib/cloud/openstack/helpers.rb:25:in `with_openstack'
/var/vcap/packages/director/gem_home/gems/bosh_openstack_cpi-1.5.0.pre.1478/lib/cloud/openstack/cloud.rb:512:in `block in set_vm_metadata'
/var/vcap/packages/director/gem_home/gems/bosh_common-1.5.0.pre.1478/lib/common/thread_formatter.rb:46:in `with_thread_name'
/var/vcap/packages/director/gem_home/gems/bosh_openstack_cpi-1.5.0.pre.1478/lib/cloud/openstack/cloud.rb:511:in `set_vm_metadata'
/var/vcap/packages/director/gem_home/gems/director-1.5.0.pre.1478/lib/director/metadata_helper.rb:14:in `update_vm_metadata'
/var/vcap/packages/director/gem_home/gems/director-1.5.0.pre.1478/lib/director/package_compiler.rb:225:in `block (2 levels) in compile_package'
/var/vcap/packages/director/gem_home/gems/director-1.5.0.pre.1478/lib/director/package_compiler.rb:290:in `prepare_vm'
/var/vcap/packages/director/gem_home/gems/director-1.5.0.pre.1478/lib/director/package_compiler.rb:224:in `block in compile_package'
/var/vcap/packages/director/gem_home/gems/director-1.5.0.pre.1478/lib/director/lock_helper.rb:89:in `block in with_compile_lock'
/var/vcap/packages/director/gem_home/gems/director-1.5.0.pre.1478/lib/director/lock.rb:58:in `lock'
/var/vcap/packages/director/gem_home/gems/director-1.5.0.pre.1478/lib/director/lock_helper.rb:88:in `with_compile_lock'
/var/vcap/packages/director/gem_home/gems/director-1.5.0.pre.1478/lib/director/package_compiler.rb:217:in `compile_package'
/var/vcap/packages/director/gem_home/gems/director-1.5.0.pre.1478/lib/director/package_compiler.rb:205:in `block (2 levels) in process_task'
/var/vcap/packages/director/gem_home/gems/director-1.5.0.pre.1478/lib/director/event_log.rb:58:in `track'
/var/vcap/packages/director/gem_home/gems/director-1.5.0.pre.1478/lib/director/package_compiler.rb:203:in `block in process_task'
/var/vcap/packages/director/gem_home/gems/bosh_common-1.5.0.pre.1478/lib/common/thread_formatter.rb:46:in `with_thread_name'
/var/vcap/packages/director/gem_home/gems/director-1.5.0.pre.1478/lib/director/package_compiler.rb:199:in `process_task'
/var/vcap/packages/director/gem_home/gems/director-1.5.0.pre.1478/lib/director/package_compiler.rb:174:in `block (4 levels) in compile_packages'
/var/vcap/packages/director/gem_home/gems/bosh_common-1.5.0.pre.1478/lib/common/thread_pool.rb:83:in `call'
/var/vcap/packages/director/gem_home/gems/bosh_common-1.5.0.pre.1478/lib/common/thread_pool.rb:83:in `block (2 levels) in create_thread'
/var/vcap/packages/director/gem_home/gems/bosh_common-1.5.0.pre.1478/lib/common/thread_pool.rb:67:in `loop'
/var/vcap/packages/director/gem_home/gems/bosh_common-1.5.0.pre.1478/lib/common/thread_pool.rb:67:in `block in create_thread'

解决办法：请查看本人博客：http://blog.csdn.net/tiger435/article/details/8808623#t5，部署成功反馈：

root@bosh-cli:~/bosh-workspace/deployments# bosh deploy
Getting deployment properties from director...
Unable to get properties list from director, trying without it...
Compiling deployment manifest...
Cannot get current deployment information from director, possibly a new deployment
Please review all changes carefully
Deploying `bosh-openstack.yml' to `microbosh-openstack' (type 'yes' to continue): yes

Director task 89

Preparing deployment
binding deployment (00:00:00)
binding releases (00:00:00)

....

Updating job nats
nats/0 (canary) (00:00:29)
Done                    1/1 00:00:29

Updating job redis
redis/0 (canary) (00:00:24)
Done                    1/1 00:00:24

Updating job postgres
postgres/0 (canary) (00:00:44)
Done                    1/1 00:00:44

Updating job powerdns
powerdns/0 (canary) (00:00:25)
Done                    1/1 00:00:25

Updating job blobstore
blobstore/0 (canary) (00:00:25)
Done                    1/1 00:00:25

Updating job director
director/0 (canary) (00:00:58)
Done                    1/1 00:00:58

Updating job registry
registry/0 (canary) (00:00:33)
Done                    1/1 00:00:33

Updating job health_monitor
health_monitor/0 (canary) (00:00:28)
Done                    1/1 00:00:28

Task 3 done
Started         2013-12-17 02:41:15 UTC
Finished        2013-12-17 02:57:08 UTC
Duration        00:15:53

Deployed `bosh-openstack.yml' to `microbosh-openstack'

8、验证部署是否成功

root@bosh-cli:~/bosh-workspace/deployments/bosh-openstack# bosh vms
Deployment `bosh-openstack'

Director task 90

Task 90 done

+------------------+---------+---------------+--------------------------+
| Job/index        | State   | Resource Pool | IPs                      |
+------------------+---------+---------------+--------------------------+
| blobstore/0      | running | common        | 50.50.0.11               |
| director/0       | running | common        | 50.50.0.10, 10.68.19.132 |
| health_monitor/0 | running | common        | 50.50.0.13               |
| nats/0           | running | common        | 50.50.0.5                |
| postgres/0       | running | common        | 50.50.0.7                |
| powerdns/0       | running | common        | 50.50.0.9, 10.68.19.131  |
| redis/0          | running | common        | 50.50.0.6                |
| registry/0       | running | common        | 50.50.0.12               |
+------------------+---------+---------------+--------------------------+

VMs total: 8

如上，所有Job的状态都是running，则部署成功。