OpenStack使用Bosh部署CloudFoundry(五)—使用Bosh部署CloudFoundry

版本说明：

CloudFoundry：V2版本-v149+（本文后续章节中的Manifest适用于149+版本的CF Release）



OpenStack：Grizzly版本+nova-network

CloudFoundry部署交流QQ群：176302388

CloudFoundry目前官方的版本为V2版，release最新发布为149，不过CloudFoundry在GitHub上的仓库代码贡献非常频繁，一般只要提出有BUG，很快就会有修改并提交，所以，最好的部署方式是直接下载源码，创建dev版的release进行部署，当然，也可以直接使用官方发布的release进行打包部署，本文中使用149.1-dev进行打包部署，如果要使用官方release，指定release路径即可，如：bosh create release release/cf-149.yml。

注意：本文中的示例Manifest仅适用于149+版本的release。

1、调整虚拟机存储配置

默认情况下blobstore/0虚拟机中/var/vcap/store的存储空间比较小，而后边上传用于部署CloudFoundry的Stemcell包和Release包时，会因为存储空间不足导致错误，所以这里先增加/var/vcap/store的挂载空间，防止发生错误。

具体步骤参看文章：http://blog.csdn.net/tiger435/article/details/8808623#t8。

说明：本环境配置文件中的所有通过Stemcell创建的虚拟机的登陆账户密码为“root/c1oudc0w”或者“vcap/c1oudc0w”

注意：即使配置挂载空间后，以后重启blobstore的虚拟机的时候一定记得再次挂载，或者可以修改为开机挂载，本人就是忘记该步骤，导致重启虚拟机后，后边部署cf-service-release时出现莫名其妙找不到package的错误，折磨几个小时。

2、下载并编译CloudFoundry发布包

依次执行以下命令从GitHub获取并更新代码

root@bosh-cli:~# mkdir -p ~/src/cloudfoundry
root@bosh-cli:~# cd ~/src/cloudfoundry
root@bosh-cli:~/src/cloudfoundry# git clone -b release-candidate git://github.com/cloudfoundry/cf-release.git
root@bosh-cli:~/src/cloudfoundry# cd cf-release
root@bosh-cli:~/src/cloudfoundry/cf-release# ./update

下载并更新完毕之后，执行以下命令创建release包，这是一个相当漫长的过程，中间还会发生多次中断，只要重新执行bosh create release --force即可：

说明：bosh create release --force 是使用最新的CloudFoundry源码创建release-dev包，可能会存在一些奇怪的BUG或者错误、问题，好处就是使用这种方式构建release-dev包，下次代码更新后，再进行打包速度会比较快。若过程中出现问题，可以使用经过测试发布的release包，打包方式：bosh create release releases/cf-149.yml，会在releases目录下生成一个tgz的压缩包，之后执行bosh
upload releases/cf-149.tgz上传release包即可。

root@bosh-cli:~/src/cloudfoundry/cf-release# bosh create release --force

....

Building jobs
-------------
Building health_manager_next...
Final version:   NOT FOUND
Dev version:     NOT FOUND
Generating...
Generated version 7.1-dev

Building dashboard...
Final version:   Downloading `dashboard (10)'...

Building ccdb_postgres...
Final version:   Downloading `ccdb_postgres (7)'...

Building gorouter...
Final version:   Downloading `gorouter (2)'...

Building dea_next...
Final version:   NOT FOUND
Dev version:     NOT FOUND
Generating...
Generated version 12.1-dev

Building collector...
Final version:   NOT FOUND
Dev version:     NOT FOUND
Generating...
Generated version 6.1-dev

Building uaa...
Final version:   NOT FOUND
Dev version:     NOT FOUND
Generating...
Generated version 27.1-dev

Building syslog_aggregator...
Final version:   NOT FOUND
Dev version:     NOT FOUND
Generating...
Generated version 10.1-dev

Building login...
Final version:   NOT FOUND
Dev version:     NOT FOUND
Generating...
Generated version 11.1-dev

Building debian_nfs_server...
Final version:   FOUND LOCAL

Building nats...
Final version:   NOT FOUND
Dev version:     NOT FOUND
Generating...
Generated version 10.1-dev

Building postgres...
Final version:   Downloading `postgres (4)'...

Building cloud_controller_ng...
Final version:   NOT FOUND
Dev version:     NOT FOUND
Generating...
Generated version 7.1-dev

Building release
----------------

Generating manifest...
----------------------
Writing manifest...

Release summary
---------------
Packages
+---------------------+----------+-------------+
| Name                | Version  | Notes       |
+---------------------+----------+-------------+
| git                 | 1        |             |
| nginx               | 9        |             |
| ruby                | 8        |             |
| health_manager_next | 18       |             |
| dashboard           | 14       |             |
| libyaml             | 1        |             |
| rootfs_lucid64      | 1        |             |
| insight_agent       | 2        |             |
| ruby_next           | 3        |             |
| gorouter            | 4.1-dev  | new version |
| dea_next            | 16.1-dev | new version |
| collector           | 8.1-dev  | new version |
| uaa                 | 24.1-dev | new version |
| golang              | 2        |             |
| warden              | 26.1-dev | new version |
| mysqlclient         | 3        |             |
| syslog_aggregator   | 4        |             |
| libpq               | 5        |             |
| login               | 13.1-dev | new version |
| dea_jvm             | 4        |             |
| buildpack_cache     | 1.1-dev  | new version |
| debian_nfs_server   | 3        |             |
| nats                | 8        |             |
| postgres            | 4        |             |
| sqlite              | 3        |             |
| imagemagick         | 2        |             |
| cloud_controller_ng | 13.1-dev | new version |
| common              | 5        |             |
+---------------------+----------+-------------+

Jobs
+---------------------+----------+-------------+
| Name                | Version  | Notes       |
+---------------------+----------+-------------+
| health_manager_next | 7.1-dev  | new version |
| dashboard           | 10       |             |
| ccdb_postgres       | 7        |             |
| gorouter            | 2        |             |
| dea_next            | 12.1-dev | new version |
| collector           | 6.1-dev  | new version |
| uaa                 | 27.1-dev | new version |
| syslog_aggregator   | 10.1-dev | new version |
| login               | 11.1-dev | new version |
| debian_nfs_server   | 7        |             |
| nats                | 10.1-dev | new version |
| postgres            | 4        |             |
| cloud_controller_ng | 7.1-dev  | new version |
+---------------------+----------+-------------+

Jobs affected by changes in this release
+---------------------+----------+
| Name                | Version  |
+---------------------+----------+
| health_manager_next | 7.1-dev  |
| dea_next            | 12.1-dev |
| collector           | 6.1-dev  |
| uaa                 | 27.1-dev |
| syslog_aggregator   | 10.1-dev |
| login               | 11.1-dev |
| nats                | 10.1-dev |
| cloud_controller_ng | 7.1-dev  |
| gorouter            | 2        |
+---------------------+----------+

Release version: 149.1-dev
Release manifest: /root/src/cloudfoundry/cf-release/dev_releases/cf-release-149.1-dev.yml

3、切换bosh target到Bosh的Director并创建新用户

root@bosh-cli:~/src/cloudfoundry/cf-release# bosh target https://10.68.19.132:25555 Target set to `bosh'
root@bosh-cli:~/src/cloudfoundry/cf-release# bosh login
Your username: admin
Enter password: *****
Logged in as `admin'
root@bosh-cli:~/src/cloudfoundry/cf-release# bosh create user
Enter new username: venusource
Enter new password: **********
Verify new password: **********
User `venusource' has been created
root@bosh-cli:~/src/cloudfoundry/cf-release# bosh login
Your username: venusource
Enter password: **********
Logged in as `venusource'

4、上传release和stemcell到Bosh中

上传Stemcell和部署Bosh的时候使用的是同一个Stemcell：

root@bosh-cli:~/src/cloudfoundry/cf-release# bosh upload stemcell ~/bosh-workspace/stemcells/bosh-stemcell-latest-openstack-kvm-ubuntu.tgz

Verifying stemcell...
File exists and readable                                     OK
Using cached manifest...
Stemcell properties                                          OK

Stemcell info
-------------
Name:    bosh-stemcell
Version: 776

Checking if stemcell already exists...
No

Uploading stemcell...

latest-bosh-s: 100% |oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo| 251.3MB  11.0MB/s Time: 00:00:22

Director task 1

Update stemcell
extracting stemcell archive (00:00:12)
verifying stemcell manifest (00:00:00)
checking if this stemcell already exists (00:00:00)
uploading stemcell bosh-stemcell/776 to the cloud (00:02:05)
save stemcell bosh-stemcell/776 (4e433167-acf2-41b0-b691-934684a63e52) (00:00:00)
Done                    5/5 00:02:17

Task 1 done
Started         2013-12-17 07:03:06 UTC
Finished        2013-12-17 07:05:23 UTC
Duration        00:02:17

Stemcell uploaded and created

上传Release到Bosh：

root@bosh-cli:~/src/cloudfoundry/cf-release# bosh upload release

....

Release has been created
cf-release/149.1-dev (00:00:00)
Done                    1/1 00:00:00

Task 2 done
Started         2013-12-18 01:59:30 UTC
Finished        2013-12-18 02:02:20 UTC
Duration        00:02:50

Release uploaded

验证下Stemcell和Release是否正常：

root@bosh-cli:~# bosh releases

+------------+------------+-------------+
| Name       | Versions   | Commit Hash |
+------------+------------+-------------+
| cf-release | 149.1-dev* | 27784f08    |
+------------+------------+-------------+
(*) Currently deployed
(+) Uncommitted changes

Releases total: 3
root@bosh-cli:~# bosh stemcells

+---------------------------+---------+--------------------------------------+
| Name                      | Version | CID                                  |
+---------------------------+---------+--------------------------------------+
| bosh-openstack-kvm-ubuntu | 1478    | 316501e1-482d-472d-9fd9-b71a7b1223ee |
+---------------------------+---------+--------------------------------------+

Stemcells total: 1

5、创建并配置用于部署CloudFoundry的Manifest文件

从GitHub中获取模板文件：

root@bosh-cli:~/src/cloudfoundry/cf-release# cd ~/bosh-workspace/deployments/
root@bosh-cli:~/bosh-workspace/deployments# vi cf-149.yml

下边给我本环境中配置后的Manifest.yml文件作为参考：

---
name: cf
director_uuid: 785aa0a6-87c7-4452-a862-c118afc90b37

releases:
- name: cf-release
version: latest

networks:
- name: floating
type: vip
cloud_properties: {}
- name: default
type: dynamic
cloud_properties:
security_groups:
- default

update:
canaries: 1
canary_watch_time: 30000-60000
update_watch_time: 30000-60000
max_in_flight: 4

compilation:
workers: 6
network: default
reuse_compilation_vms: true
cloud_properties:
instance_type: v1.medium

resource_pools:
- name: small
network: default
size: 4
stemcell:
name: bosh-openstack-kvm-ubuntu
version: latest
cloud_properties:
instance_type: v1.small

- name: medium
network: default
size: 0
stemcell:
name: bosh-openstack-kvm-ubuntu
version: latest
cloud_properties:
instance_type: v1.medium

jobs:
- name: data
release: cf-release
template:
- postgres
- debian_nfs_server
instances: 1
resource_pool: small
persistent_disk: 4096
networks:
- name: default
default:
- dns
- gateway
properties:
db: databases

- name: core
release: cf-release
template:
- syslog_aggregator
- nats
- health_manager_next
- uaa
instances: 1
resource_pool: small
networks:
- name: default
default:
- dns
- gateway

- name: api
release: cf-release
template:
- cloud_controller_ng
- gorouter
instances: 1
resource_pool: small
networks:
- name: default
default:
- dns
- gateway
- name: floating
static_ips:
- 10.68.19.133
properties:
db: databases

- name: dea
release: cf-release
template:
- dea_next
instances: 1
resource_pool: small
networks:
- name: default
default: [dns, gateway]

properties:
cf:
name: cf
dns: vsc.com
ip_addresses: ["10.68.19.133"]
deployment_size: medium
security_group: default
persistent_disk: 4096
common_password: c1oudc0w
dea_server_ram: 4096

domain: vsc.com
system_domain: vsc.com.com
system_domain_organization: system_domain
app_domains:
- vsc.com

networks:
apps: default
management: default

nats:
address: 0.core.default.cf.bosh
machines:
- 0.core.default.cf.bosh
port: 4222
user: nats
password: c1oudc0w
authorization_timeout: 5

router:
address: 0.api.default.cf.bosh
port: 8081
status:
port: 8080
user: gorouter
password: c1oudc0w

dea: &dea
memory_mb: 4096
disk_mb: 10240
directory_server_protocol: http

dea_next: *dea

syslog_aggregator:
address: 0.core.default.cf.bosh
port: 54321

nfs_server:
address: 0.data.default.cf.bosh
network: "*.cf.bosh"
idmapd_domain: vsc.com

debian_nfs_server:
no_root_squash: true

databases: &databases
db_scheme: postgres
address: 0.data.default.cf.bosh
port: 5524
roles:
- tag: admin
name: ccadmin
password: c1oudc0w
- tag: admin
name: uaaadmin
password: c1oudc0w
databases:
- tag: cc
name: ccdb
citext: true
- tag: uaa
name: uaadb
citext: true

ccdb: &ccdb
db_scheme: postgres
address: 0.data.default.cf.bosh
port: 5524
roles:
- tag: admin
name: ccadmin
password: c1oudc0w
databases:
- tag: cc
name: ccdb
citext: true

ccdb_ng: *ccdb

uaadb:
db_scheme: postgresql
address: 0.data.default.cf.bosh
port: 5524
roles:
- tag: admin
name: uaaadmin
password: c1oudc0w
databases:
- tag: uaa
name: uaadb
citext: true

cc_api_version: v2

cc: &cc
logging_level: debug
external_host: api
srv_api_uri: http://api.vsc.com cc_partition: default
db_encryption_key: c1oudc0w
bootstrap_admin_email: admin@vsc.com
bulk_api_password: c1oudc0w
uaa_resource_id: cloud_controller
staging_upload_user: uploaduser
staging_upload_password: c1oudc0w
resource_pool:
resource_directory_key: cc-resources
# Local provider when using NFS
fog_connection:
provider: Local
packages:
app_package_directory_key: cc-packages
droplets:
droplet_directory_key: cc-droplets
default_quota_definition: runaway

ccng: *cc

login:
enabled: false

uaa:
url: http://uaa.vsc.com spring_profiles: postgresql
no_ssl: true
catalina_opts: -Xmx768m -XX:MaxPermSize=256m
resource_id: account_manager
jwt:
signing_key: |
-----BEGIN RSA PRIVATE KEY-----
MIICXAIBAAKBgQDHFr+KICms+tuT1OXJwhCUmR2dKVy7psa8xzElSyzqx7oJyfJ1
JZyOzToj9T5SfTIq396agbHJWVfYphNahvZ/7uMXqHxf+ZH9BL1gk9Y6kCnbM5R6
0gfwjyW1/dQPjOzn9N394zd2FJoFHwdq9Qs0wBugspULZVNRxq7veq/fzwIDAQAB
AoGBAJ8dRTQFhIllbHx4GLbpTQsWXJ6w4hZvskJKCLM/o8R4n+0W45pQ1xEiYKdA
Z/DRcnjltylRImBD8XuLL8iYOQSZXNMb1h3g5/UGbUXLmCgQLOUUlnYt34QOQm+0
KvUqfMSFBbKMsYBAoQmNdTHBaz3dZa8ON9hh/f5TT8u0OWNRAkEA5opzsIXv+52J
duc1VGyX3SwlxiE2dStW8wZqGiuLH142n6MKnkLU4ctNLiclw6BZePXFZYIK+AkE
xQ+k16je5QJBAN0TIKMPWIbbHVr5rkdUqOyezlFFWYOwnMmw/BKa1d3zp54VP/P8
+5aQ2d4sMoKEOfdWH7UqMe3FszfYFvSu5KMCQFMYeFaaEEP7Jn8rGzfQ5HQd44ek
lQJqmq6CE2BXbY/i34FuvPcKU70HEEygY6Y9d8J3o6zQ0K9SYNu+pcXt4lkCQA3h
jJQQe5uEGJTExqed7jllQ0khFJzLMx0K6tj0NeeIzAaGCQz13oo2sCdeGRHO4aDh
HH6Qlq/6UOV5wP8+GAcCQFgRCcB+hrje8hfEEefHcFpyKH+5g1Eu1k0mLrxK2zd+
4SlotYRHgPCEubokb2S1zfZDWIXW3HmggnGgM949TlY=
-----END RSA PRIVATE KEY-----
verification_key: |
-----BEGIN PUBLIC KEY-----
MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDHFr+KICms+tuT1OXJwhCUmR2d
KVy7psa8xzElSyzqx7oJyfJ1JZyOzToj9T5SfTIq396agbHJWVfYphNahvZ/7uMX
qHxf+ZH9BL1gk9Y6kCnbM5R60gfwjyW1/dQPjOzn9N394zd2FJoFHwdq9Qs0wBug
spULZVNRxq7veq/fzwIDAQAB
-----END PUBLIC KEY-----
cc:
client_secret: c1oudc0w
admin:
client_secret: c1oudc0w
batch:
username: batchuser
password: c1oudc0w
client:
autoapprove:
- cf
clients:
cf:
override: true
authorized-grant-types: password,implicit,refresh_token
authorities: uaa.none
scope: cloud_controller.read,cloud_controller.write,openid,password.write,cloud_controller.admin,scim.read,scim.write
access-token-validity: 7200
refresh-token-validity: 1209600
scim:
users:
- admin|c1oudc0w|scim.write,scim.read,openid,cloud_controller.admin
- services|c1oudc0w|scim.write,scim.read,openid,cloud_controller.admin

修改cf.yml文件

director_uuid：修改为Bosh Director的UUID，可以通过bosh status查看；

releases的name属性，修改为之前创建CloudFoundry的Release包时输入的包名；

10.58.19.133：修改为可用的Floating IP；

vsc.com：域名，CF平台不能通过IP直接访问使用，必须配置域名，可以自己任意写个域名，然后配置本地DNS Server进行解析即可；

反馈：

root@bosh-cli:~/bosh-workspace/deployments# bosh deployment cf-149.yml

root@bosh-cli:~/bosh-workspace/deployments# bosh deploy
Getting deployment properties from director...
Unable to get properties list from director, trying without it...
Compiling deployment manifest...
Cannot get current deployment information from director, possibly a new deployment
Please review all changes carefully
Deploying `cf-149.yml' to `bosh' (type 'yes' to continue): yes

Director task 10

Preparing deployment
binding deployment (00:00:00)
binding releases (00:00:00)
binding existing deployment (00:00:00)
binding resource pools (00:00:00)
binding stemcells (00:00:00)
binding templates (00:00:01)
binding properties (00:00:00)
binding unallocated VMs (00:00:00)
binding instance networks (00:00:00)
Done                    9/9 00:00:01

Preparing package compilation

Preparing DNS
binding DNS (00:00:00)
Done                    1/1 00:00:00

Creating bound missing VMs
small/0 (00:00:42)
small/1 (00:00:57)
small/2 (00:01:02)
small/3 (00:01:10)
Done                    4/4 00:01:10

Binding instance VMs
data/0 (00:00:01)
dea/0 (00:00:01)
core/0 (00:00:01)
api/0 (00:00:01)
Done                    4/4 00:00:01

Preparing configuration
binding configuration (00:00:02)
Done                    1/1 00:00:02

Updating job data
data/0 (canary) (00:00:53)
Done                    1/1 00:00:53

Updating job core
core/0 (canary) (00:01:08)
Done                    1/1 00:01:08

Updating job api
api/0 (canary) (00:01:04)
Done                    1/1 00:01:04

Updating job dea
dea/0 (canary) (00:03:20)
Done                    1/1 00:03:20

Task 10 done
Started         2013-12-18 11:20:53 UTC
Finished        2013-12-18 11:29:37 UTC
Duration        00:08:44

Deployed `cf-149.yml' to `bosh'

部署过程可能会因为OpenStack的VM虚拟机创建时间过长或PING虚拟机响应过长导致失败，只需要删除Deployments后再重新部署即可。

Deployment删除命令：

root@bosh-cli:~/bosh-workspace/deployments# bosh delete deployment cf

部署成功之后可以使用命令查看虚拟机运行状态：

root@bosh-cli:~/bosh-workspace/deployments# bosh vms
Deployment `cf'

Director task 191

Task 191 done

+-----------+---------+---------------+--------------------------+
| Job/index | State   | Resource Pool | IPs                      |
+-----------+---------+---------------+--------------------------+
| api/0     | running | small         | 50.50.0.16, 10.68.19.133 |
| core/0    | running | small         | 50.50.0.17               |
| data/0    | running | small         | 50.50.0.14               |
| dea/0     | running | small         | 50.50.0.15               |
+-----------+---------+---------------+--------------------------+

VMs total: 4

全部Job为Running状态则正常。

6、使用CF客户端管理CloudFoundry运行环境

注意：请先配置DNS Server，对设置的虚拟域名（vsc.com）进行解析，参照文章：《配置DNS Server》和《配置OpenStack环境DNS》

6.1、安装CF命令行工具

gem install cf

6.2、切换CF指向为api.vsc.com

root@bosh-cli:~# cf target api.vsc.com
Setting target to http://api.vsc.com... OK

target: http://api.vsc.com organization: system_domain

6.3、登陆到CF平台，本环境账号密码为admin/c1oudc0w

root@bosh-cli:~# cf login admin
target: http://api.vsc.com 
Password> ********

Authenticating... OK
There are no spaces. You may want to create one with create-space.

6.4、创建组织

root@bosh-cli:~# cf create-org venusource
Creating organization venusource... OK
Switching to organization venusource... OK
There are no spaces. You may want to create one with create-space.

6.5、创建并切换空间

root@bosh-cli:~# cf create-space development
Creating space development... OK
Adding you as a manager... OK
Adding you as a developer... OK
Space created! Use `cf switch-space development` to target it.
root@bosh-cli:~# cf switch-space development
Switching to space development... OK

target: http://api.vsc.com organization: venusource
space: development

6.6、下载测试Ruby工程

root@bosh-cli:~# git clone https://github.com/cloudfoundry-community/cf_demoapp_ruby_rack.git[/code]6.7、打包并发布测试工程 
root@bosh-cli:~# cd cf_demoapp_ruby_rack/
root@bosh-cli:~/cf_demoapp_ruby_rack# rm manifest.yml
root@bosh-cli:~/cf_demoapp_ruby_rack# bundle
Resolving dependencies...
Using rack (1.5.2)
Using bundler (1.3.5)
Your bundle is complete!
Use `bundle show [gemname]` to see where a bundled gem is installed.
root@bosh-cli:~/cf_demoapp_ruby_rack# cf push
Name> hello

Instances> 1

1: 128M
2: 256M
3: 512M
4: 1G
Memory Limit> 1

Creating hello... OK

1: hello
2: none
Subdomain> hello

1: vsc.com
2: none
Domain> mycloud.com

Creating route hello.mycloud.com... OK
Binding hello.mycloud.com to hello... OK

Create services for application?> n

Save configuration?> n

Uploading hello... OK
Starting hello... OK
-----> Downloaded app package (4.0K)

....

Checking hello...
Staging in progress...
0/1 instances: 1 starting
0/1 instances: 1 starting
0/1 instances: 1 starting
1/1 instances: 1 running
OK


部署完毕！欢迎各位交流指正，谢谢！