未加星标

在Swarm上部署Harbor

字体大小 | |
[系统(linux) 所属分类 系统(linux) | 发布者 店小二04 | 时间 20160901 | 作者 红领巾 ] 0人收藏点击收藏

Harbor作为一个企业级的Registry服务,单机的部署并不具备故障转移等功能。将harbor部署到Docker Swarm集群上能够使harbor实现一个简单的高可用性。Docker Swarm部署简单,直接与Docker Daemon通信,我们只要使用swarm版的docker-compose.yml,就能仍然使用原有的docker-compose的命令完成harbor的部署。

搭建Docker Swarm集群
部署环境
swarm-manager/consul: 10.117.7.234
swarm-node01: 10.117.7.164
swarm-node02: 10.117.4.251
swarm-node03: 10.117.5.196
四台机器:Ubuntu15.10/kernel:4.2.0-16-generic
注意:
由于需要在集群上部署overlay网络,因此搭建集群选用的宿主机的kenerl必须大于3.16。
部署consul服务
docker run -d --restart=always -p 8500:8500 --name=consul progrium/consul -server -bootstrap
部署swarm-manager节点
//此处host_ip=10.117.7.234
docker run -d -p 4000:4000 --restart=always --name --experimental swarm-manager swarm manage -H :4000 --replication --advertise <host_ip>:4000 consul://<host_ip>:8500
部署swarm-node节点
//此处以swarm-node01节点为例
//swarm-manager_ip=10.117.7.234
//host_ip=10.117.7.164
docker run -d --name swarm-node01 --restart=always --experimental swarm join --advertise=<swarm-manager_ip>:2375 consul://<host_ip>:8500
注意:
由于需要swarm集群能够支持container的故障转移,因此在部署swarm-manager和swarm-node时都需要加上–experimental的参数。

此时我们可以看到swarm集群已搭建完成,各个节点也都处于健康(Healthy)状态。

[email protected]:/home/work/harbor/Deploy# docker -H :4000 info
Containers: 3
Running: 3
Paused: 0
Stopped: 0
Images: 21
Server Version: swarm/1.2.4
Role: primary
Strategy: spread
Filters: health, port, containerslots, dependency, affinity, constraint
Nodes: 3
node-01: 10.117.7.164:2375
└ ID: ME36:X754:PUJI:GIM2:LSYX:U5BW:55E6:7EWZ:IMNU:CTR2:M4XU:PF3L
└ Status: Healthy
└ Containers: 1 (1 Running, 0 Paused, 0 Stopped)
└ Reserved CPUs: 0 / 1
└ Reserved Memory: 0 B / 1.018 GiB
└ Labels: kernelversion=4.2.0-16-generic, operatingsystem=Ubuntu 15.10, storagedriver=aufs
└ UpdatedAt: 2016-09-06T06:19:47Z
└ ServerVersion: 1.11.2
node-02: 10.117.4.251:2375
└ ID: WBD3:2Q7Y:PI5N:2ZMN:IFFW:7P7A:46OE:GCX7:6URC:OOOQ:ETBM:OY4H
└ Status: Healthy
└ Containers: 1 (1 Running, 0 Paused, 0 Stopped)
└ Reserved CPUs: 0 / 1
└ Reserved Memory: 0 B / 1.018 GiB
└ Labels: kernelversion=4.2.0-16-generic, operatingsystem=Ubuntu 15.10, storagedriver=aufs
└ UpdatedAt: 2016-09-06T06:19:28Z
└ ServerVersion: 1.11.2
node-03: 10.117.5.196:2375
└ ID: PU6M:VI4X:MYHR:EURV:INHG:FXR5:VNOW:AMVC:UL66:4UK2:Z2TU:HMVK
└ Status: Healthy
└ Containers: 1 (1 Running, 0 Paused, 0 Stopped)
└ Reserved CPUs: 0 / 1
└ Reserved Memory: 0 B / 1.018 GiB
└ Labels: kernelversion=4.2.0-16-generic, operatingsystem=Ubuntu 15.10, storagedriver=aufs
└ UpdatedAt: 2016-09-06T06:19:40Z
└ ServerVersion: 1.11.2
Plugins:
Volume:
Network:
Kernel Version: 4.2.0-16-generic
Operating System: linux
Architecture: amd64
CPUs: 3
Total Memory: 3.054 GiB
Name: 8865dfec9652
Docker Root Dir:
Debug mode (client): false
Debug mode (server): false
WARNING: No kernel memory limit support
Experimental: true
挂载nfs共享存储
为了保证swarm各个节点上数据的一致性,我们需要在各节点上挂载nfs共享存储。
此处,我们以swarm-manager作为nfs服务器端,以其他node节点作为客户端。nfs挂载的详细步骤在此不做赘述。
nfs服务器端(swarm-manager)
[email protected]:~# df -h
Filesystem Size Used Avail Use% Mounted on
udev 481M 0 481M 0% /dev
tmpfs 100M 13M 87M 14% /run
/dev/dm-0 31G 2.1G 27G 8% /
tmpfs 497M 36K 497M 1% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 497M 0 497M 0% /sys/fs/cgroup
/dev/sda1 236M 51M 173M 23% /boot
10.117.7.234:/home/work/harbor 31G 2.1G 27G 8% /mnt/nfs/harbor
tmpfs 100M 0 100M 0% /run/user/0
nfs客户端(swam-node)
[email protected]:~$ df -h
Filesystem Size Used Avail Use% Mounted on
udev 481M 0 481M 0% /dev
tmpfs 100M 14M 87M 14% /run
/dev/dm-0 31G 3.7G 25G 13% /
tmpfs 497M 200K 497M 1% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 497M 0 497M 0% /sys/fs/cgroup
/dev/sda1 236M 51M 173M 23% /boot
10.117.7.234:/home/work/harbor 31G 2.1G 27G 8% /home/work/harbor
tmpfs 100M 0 100M 0% /run/user/1000
部署harbor
在swarm-manager节点的harbor共享目录下,利用docker-compose启动harbor即可
[email protected]:/home/work/harbor/Deploy# docker-compose -H : 4000 -f docker-compose.swarm.yml up -d
注意:
- 请使用docker-compose.swarm.yml
- 必须使用daemon的方式启动,否则可能启动失败。

此时我们可以看到harbor的各个容器模块已经部署到了集群的各个节点上。

[email protected]:/home/work/harbor/Deploy# docker -H :4000 ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
8cb654632664 deploy_jobservice "/go/bin/harbor_jobse" 7 seconds ago Up 5 seconds node-01/deploy_jobservice_1
e2b1bd9b6b7e nginx:1.9 "nginx -g 'daemon off" 7 seconds ago Up 6 seconds 10.117.4.251:80->80/tcp, 10.117.4.251:443->443/tcp node-02/deploy_proxy_1
92ee66a7d13d deploy_ui "/go/bin/harbor_ui" 8 seconds ago Up 6 seconds 80/tcp node-02/deploy_ui_1
20deddc1b297 registry:2.4.0 "/bin/registry serve " 12 seconds ago Up 7 seconds 5000/tcp, 10.117.5.196:5001->5001/tcp node-03/deploy_registry_1
fa6f151e06f9 deploy_mysql "docker-entrypoint.sh" 12 seconds ago Up 9 seconds 3306/tcp node-01/deploy_mysql_1
41cf93197e8b deploy_log "/bin/sh -c 'cron && " 13 seconds ago Up 12 seconds 10.117.4.251:1514->514/tcp node-02/deploy_log_1
高可用性测试
接下来我们测试一下swarm是否能够对harbor容器进行故障转移。

由以上部署我们可以看到harbor的registry模块是部署在node03节点上的。我们将node03节点重启。

[email protected]:/home/work# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
20deddc1b297 registry:2.4.0 "/bin/registry serve " 2 minutes ago Up 2 minutes 5000/tcp, 0.0.0.0:5001->5001/tcp deploy_registry_1
cc107ad68270 swarm "/swarm --experimenta" 3 weeks ago Up 3 weeks 2375/tcp swarm-node03
[email protected]:/home/work# reboot
通过docker info命令我们可以看到,此时node03节点处于Unhealty状态。
[email protected]:/home/work/harbor/Deploy# docker -H :4000 info
Containers: 9
Running: 8
Paused: 0
Stopped: 1
Images: 21
Server Version: swarm/1.2.4
Role: primary
Strategy: spread
Filters: health, port, containerslots, dependency, affinity, constraint
Nodes: 3
node-01: 10.117.7.164:2375
└ ID: ME36:X754:PUJI:GIM2:LSYX:U5BW:55E6:7EWZ:IMNU:CTR2:M4XU:PF3L
└ Status: Healthy
└ Containers: 4 (4 Running, 0 Paused, 0 Stopped)
└ Reserved CPUs: 0 / 1
└ Reserved Memory: 0 B / 1.018 GiB
└ Labels: kernelversion=4.2.0-16-generic, operatingsystem=Ubuntu 15.10, storagedriver=aufs
└ UpdatedAt: 2016-09-06T06:44:59Z
└ ServerVersion: 1.11.2
node-02: 10.117.4.251:2375
└ ID: WBD3:2Q7Y:PI5N:2ZMN:IFFW:7P7A:46OE:GCX7:6URC:OOOQ:ETBM:OY4H
└ Status: Healthy
└ Containers: 4 (4 Running, 0 Paused, 0 Stopped)
└ Reserved CPUs: 0 / 1
└ Reserved Memory: 0 B / 1.018 GiB
└ Labels: kernelversion=4.2.0-16-generic, operatingsystem=Ubuntu 15.10, storagedriver=aufs
└ UpdatedAt: 2016-09-06T06:44:29Z
└ ServerVersion: 1.11.2
node-03: 10.117.5.196:2375
└ ID: PU6M:VI4X:MYHR:EURV:INHG:FXR5:VNOW:AMVC:UL66:4UK2:Z2TU:HMVK
└ Status: Unhealthy
└ Containers: 1
└ Reserved CPUs: 0 / 1
└ Reserved Memory: 0 B / 1.018 GiB
└ Labels: kernelversion=4.2.0-16-generic, operatingsystem=Ubuntu 15.10, storagedriver=aufs
└ Error: Cannot connect to the Docker daemon. Is the docker daemon running on this host?
└ UpdatedAt: 2016-09-06T06:44:48Z
└ ServerVersion: 1.11.2

通过docker logs swarm-manager查询swarm-manager的日志,如下,我们发现swarm-manager将故障节点node03上的registry容器重新调度到了健康的node01节点上。

time="2016-09-06T06:44:06Z" level=info msg="Rescheduled container c7d868a2dabcd13b27e01c674cfe1152e5358aeb69a50e4a2ff0f2d0b55f3464 from node-03 to node-01 as b731057db26bc72eda8b76af1e7bac2989c9d2458b637f7a61e741536a462e05"
time="2016-09-06T06:44:06Z" level=info msg="Container c7d868a2dabcd13b27e01c674cfe1152e5358aeb69a50e4a2ff0f2d0b55f3464 was running, starting container b731057db26bc72eda8b76af1e7bac2989c9d2458b637f7a61e741536a462e05"

可以看出,registry容器被重新调度到了node01节点上,harbor仍然能够正常访问。

[email protected]:/home/work/harbor/Deploy# docker -H :4000 ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
b731057db26b registry:2.4.0 "/bin/registry serve " 36 seconds ago Up 33 seconds 5000/tcp, 10.117.7.164:5001->5001/tcp node-01/deploy_registry_1
25f1d75ec18a deploy_jobservice "/go/bin/harbor_jobse" 5 minutes ago Up 5 minutes node-01/deploy_jobservice_1
f8c2c7fdb24d nginx:1.9 "nginx -g 'daemon off" 5 minutes ago Up 5 minutes 10.117.4.251:80->80/tcp, 10.117.4.251:443->443/tcp node-02/deploy_proxy_1
55862d1306f8 deploy_ui "/go/bin/harbor_ui" 5 minutes ago Up 5 minutes 80/tcp node-01/deploy_ui_1
0fa10a432777 deploy_mysql "docker-entrypoint.sh" 5 minutes ago Up 5 minutes 3306/tcp node-02/deploy_mysql_1
bb36d2f15f52 deploy_log "/bin/sh -c 'cron && " 5 minutes ago Up 5 minutes 10.117.4.251:1514->514/tcp node-02/deploy_log_1

注意:要使swarm能够完成container的转移,必须保证各个节点上都预先build好了harbor所需镜像。

本文地址:http://www.codesec.net/Linux/2016-09/134954.htm


在Swarm上部署Harbor

本文系统(linux)相关术语:linux系统 鸟哥的linux私房菜 linux命令大全 linux操作系统

主题: CPUUbuntuDocker服务器SwarmLinux数据5G
分页:12
转载请注明
本文标题:在Swarm上部署Harbor
本站链接:http://www.codesec.net/view/481893.html
分享请点击:


1.凡CodeSecTeam转载的文章,均出自其它媒体或其他官网介绍,目的在于传递更多的信息,并不代表本站赞同其观点和其真实性负责;
2.转载的文章仅代表原创作者观点,与本站无关。其原创性以及文中陈述文字和内容未经本站证实,本站对该文以及其中全部或者部分内容、文字的真实性、完整性、及时性,不作出任何保证或承若;
3.如本站转载稿涉及版权等问题,请作者及时联系本站,我们会及时处理。
登录后可拥有收藏文章、关注作者等权限...
技术大类 技术大类 | 系统(linux) | 评论(0) | 阅读(30)