MySQL、MongoDB、Redis数据库Docker镜像制作
2016-10-22 16:23
736 查看
MySQL、MongoDB、Redis数据库Docker镜像制作
在多台主机上进行数据库部署时,如果使用传统的MySQL的交互式的安装方式将会重复很多遍。如果做成镜像,那么我们只需要make once,就run everywhere了。
本文的Dockerfile内容包括MySQL、MongoDB、Redis、JDK、FastDFS
Dockerfile样例
下面的样例中,在运行前需要将所需的软件包或文件夹放到tar目录下,其中还包括已经提前编译好的FastDFS。指定了数据库需要的存储文件夹(镜像之外的可挂载的目录)以及容器运行时对外开放的端口。还指定了容器启动后执行的脚本:entrypoint.sh#Dockerfile for my specific purpose:ZH-Dist-Crawler #pull base image FROM ubuntu:14.04 #MAINTAINER abc #the default user is root #USER root #add specific source to /etc/apt/sources.list mirrors.163.com RUN echo 'deb http://mirrors.163.com/ubuntu/ trusty main restricted universe multiverse' > /etc/apt/sources.list &&\ echo 'deb http://mirrors.163.com/ubuntu/ trusty-updates main restricted universe multiverse' >> /etc/apt/sources.list #install some basic tools:ssh wget/curl .etc # Remove unneeded /var/lib/apt/lists/* after install to reduce the docker image size (tens or hundrands of megabytes) # -qq used for hide apt-get's output RUN apt-get -qqy update ;\ #apt-get install -y openssh-server wget ;\ apt-get -y install gcc make #NOTICE:comment must be in seperate line #create directories RUN \ #mkdir -p /var/run/sshd ;\ #mkdir -p /root/.ssh ;\ #for saving dump.rdb and read redis.conf mkdir -p /data/redis &&\ mkdir -p /data/mongo/db /data/mongo/log &&\ #FastDFS #tmp dir will be removed mkdir -p /tmp/fdfs &&\ mkdir -p /data/fdfs/tracker /data/fdfs/storage &&\ #mkdir -p /etc/mysql &&\ #when install mysql from ubuntu source,it will be created mkdir -p /opt/redis &&\ mkdir -p /opt/mongo #install mysql from apt source #RUN \ # apt-get update && \ # DEBIAN_FRONTEND=noninteractive apt-get install -y mysql-server && \ # rm -rf /var/lib/apt/lists/* && \ # sed -i 's/^\(bind-address\s.*\)/# \1/' /etc/mysql/my.cnf && \ # sed -i 's/^\(log_error\s.*\)/# \1/' /etc/mysql/my.cnf && \ # echo "mysqld_safe &" > /tmp/config && \ # echo "mysqladmin --silent --wait=30 ping || exit 1" >> /tmp/config && \ # echo "mysql -e 'GRANT ALL PRIVILEGES ON *.* TO \"root\"@\"%\" WITH GRANT OPTION;'" >> /tmp/config && \ # bash /tmp/config && \ # rm -f /tmp/config #add files to image:redis mongodb fastdfs #COPY is same as 'ADD' but without the tar and remote url handling.recomment to use COPY COPY tar/fdfs /tmp/fdfs COPY tar/redis /opt/redis COPY tar/mongo /opt/mongo #environment for reference from bootstrap.sh to start the servers ENV REDIS_CONFIG /data/redis/redis.conf #ENV MONGO_CONFIG /data/mongo/mongodb.conf #ENV MYSQL_CONFIG /data/mysql/my.cnf ENV FDFS_TRACKER_CONFIG /data/fdfs/tracker.conf ENV FDFS_STORAGE_CONFIG /data/fdfs/storage.conf #tar/fdfs directory should include the compiled x64 files,will be installed to /usr/bin #install FastDFS and move configs to the /data/xxx directory,and this directory may also include data #can be override by add the COLUME. #WORKDIR /tmp/fdfs RUN \ #install to /usr/bin cd /tmp/fdfs/libfastcommon &&\ ./make.sh install &&\ cd /tmp/fdfs/FastDFS &&\ ./make.sh install &&\ mv /tmp/fdfs/*.conf /data/fdfs/ &&\ rm -rf /tmp/fdfs &&\ #other configs mv /opt/redis/redis.conf $REDIS_CONFIG #specify the version ENV JAVA_MAJOR 7 ENV JAVA_MINOR 80 ENV JAVA_HOME /opt/java/jdk1.$JAVA_MAJOR.0_$JAVA_MINOR COPY tar/jdk-"$JAVA_MAJOR"u$JAVA_MINOR-linux-x64.tar.gz /jdk.tgz RUN \ JDK_FILE=/jdk.tgz &&\ mkdir -p /opt/java &&\ tar -xzf $JDK_FILE -C /opt/java &&\ chmod ugo+x $JAVA_HOME/bin/* &&\ rm $JDK_FILE #install mysql from binary local file COPY tar/mysql-5.5.49-linux2.6-x86_64.tar.gz /mysql.tgz # add our user and group first to make sure their IDs get assigned consistently, regardless of whatever dependencies get added RUN groupadd -r mysql && useradd -r -g mysql mysql #libaio.so.1 is needed by mysqld to run RUN \ AR=/mysql.tgz \ && mkdir -p /usr/local/mysql \ && tar -xzf $AR -C /usr/local/mysql --strip-components=1 \ && rm $AR \ && rm -rf /usr/local/mysql/mysql-test /usr/local/mysql/sql-bench \ && rm -rf /usr/local/mysql/bin/*-debug /usr/local/mysql/bin/*_embedded \ && find /usr/local/mysql -type f -name "*.a" -delete \ && apt-get -y update && apt-get install -y libaio1 binutils && rm -rf /var/lib/apt/lists/* \ && { find /usr/local/mysql -type f -executable -exec strip --strip-all '{}' + || true; } \ && apt-get purge -y --auto-remove binutils gcc make \ && apt-get clean #change timezone to UTC+8 ,-f means force replace if exist destination file RUN ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime # replicate some of the way the APT package configuration works,create running state dir before running mysqld,or will show error:Bind on unix socket: No such file or directory RUN mkdir -p /etc/mysql/conf.d \ && { \ echo '[mysqld]'; \ echo 'skip-host-cache'; \ echo 'skip-name-resolve'; \ echo 'user = mysql'; \ echo 'datadir = /var/lib/mysql'; \ echo '!includedir /etc/mysql/conf.d/'; \ } > /etc/mysql/my.cnf \ && mkdir -p /var/run/mysqld \ && chown mysql:mysql /var/run/mysqld # can remove some pre-installed softwares such as python3 ENV REDIS /opt/redis ENV MONGO /opt/mongo # PATH ENV PATH $PATH:$JAVA_HOME/bin:$REDIS:$MONGO/bin:/usr/local/mysql/bin:/usr/local/mysql/scripts #define mountable directories VOLUME ["/data/mongo"] VOLUME ["/etc/mysql/conf.d", "/var/lib/mysql"] VOLUME ["/data/redis"] VOLUME ["/data/fdfs"] COPY bootstrap.sh /entrypoint.sh COPY mysql-entrypoint.sh /mysql-entrypoint.sh #will always be excuted and can only be excuted in container ENTRYPOINT ["/entrypoint.sh"] # Define default command. #CMD ["mysqld_safe"] #CMD is as params to ENTRYPOINT and can be override when run the image #expose ports,the host can be equal to this,better to specific a different one,such as 6379+10001 # redis #mysql #mongodb #fdfs_tracker #fdfs_storage EXPOSE 6379 3306 27017 22122 23000
启动脚本entrypoint.sh
#!/bin/bash #run by docker ENTRYPOINYT #NOTE that this script does not call interactive shell at last,so you must run 'docker exec <id or name> bash' to get the shell,and type Ctrl+D to exit that bash,but has no relation to this script's signal handling # trap SIGTERM and call terminate() ,INT for Ctrl+C trap terminate TERM trap terminate INT #####!!!WARNING!!!##### #MUST MAKE SURE THAT THIS SCIRPT IS RUN AS PID 1, docker will send SIGTERM to pid 1 process only.Luckily, when run as ENTRYPOINT,its pid is 1,but shell would not forward signals to its children. #The optional way is run Supervisord as pid 1 and use it to manage daemon services. function terminate() { #should set a long time between SIGTERM and SIGKILL by docker stop ----time=30 (the default inteval is 10s) echo "** Trapped SIGTERM(-15) or INT(-2),stopping DBs..." # MONGOPID=`ps -ef | grep 'mongod' | grep -v grep | awk '{print $2}'` # if [ ! -z $MONGOPID ]; then #not null then send signal,kill default send -15(-TERM) # kill -15 $MONGOPID # fi #killall send signal to process by name,if success no output,else output info to stderr #killall: command not found,reason:in ubuntu:14.04 docker image,it does not include the cmd,use pkill instead pkill mysqld #-15 pkill redis-server #Perform a blocking SAVE if at least one save point is configured. #if configured saving ratio and directory,kill -15 redis-server is same as call: # redis-cli shutdown save pkill fdfs_trackerd 2> /dev/null pkill fdfs_storaged 2> /dev/null pkill mongod #the signal handling is asynchronized (multi process),so wait some time in the main process in case that the mongd terminate early than redis which is in other process. #when main process exit without wait children,the docker container will exit,forcelly kill other process #for that I have put 'wait' cmd in the last,there is no need to sleep a period of time #sleep 5 } function show_usage() { echo "Usage: $0 {master|slave} " #specIP } function start_mongo_bg() { echo "Starting MongoDB server" ulimit -v unlimited ulimit -n 100000 #echo never > /sys/kernel/mm/transparent_hugepage/enabled #should be run in host #echo never > /sys/kernel/mm/transparent_hugepage/defrag #--fork seems to fork tweence mongod --quiet --dbpath=/data/mongo/db --logpath=/data/mongo/log/mongod.log & # --port 37017 --rest } function start_mysql_bg() { echo 'starting mysql' /mysql-entrypoint.sh mysqld $MYSQL_PARAM & } function start_redis_bg() { #sysctl vm.overcommit_memory=1 #should be run in host #no need to add '&' when set daemonize yes redis-server $REDIS_CONFIG } #not used by now,maybe block the terminate() echo display function looping() { while :; do sleep 300 done } if [ $# -lt 1 ];then show_usage exit 0 fi #add ip to eth0, set --cap-add=NET_ADMIN when use command 'docker run' #echo "adding ip:$2 to eth0" #/sbin/ip addr add $2 dev eth0 #param can be master/slave case "$1" in master) echo 'start master dbs' fdfs_trackerd $FDFS_TRACKER_CONFIG start_mysql_bg start_redis_bg start_mongo_bg #run in background #must use wait to wait children,or would not receive signal and handle through terminate() echo 'waiting chidren exit...' wait echo 'docker container exit' ;; slave) echo 'start slave' fdfs_storaged $FDFS_STORAGE_CONFIG start_mongo_bg echo 'waiting chidren exit...' wait echo 'docker container exit' ;; *) echo 'Usage: $0 {master|slave}' exit 1 esac #if the main thread exit and docker container will exit,so need to prevent it run as daemon #looping
mysql-entrypoint.sh
#!/bin/bash set -eo pipefail #$@ is all of the parameters passed to the script : mysqld -xxx # if command starts with an option, prepend mysqld,${VAR:start0:num} return a substr if [ "${1:0:1}" = '-' ]; then set -- mysqld "$@" fi # skip setup if they want an option that stops mysqld wantHelp= for arg; do case "$arg" in -'?'|--help|--print-defaults|-V|--version) wantHelp=1 break ;; esac done if [ ! -z "$wantHelp" ]; then echo 'if you want to store the db data when container exit,you must use -v /path/externalDir:/var/lib/mysql' echo 'else specify one of MYSQL_ROOT_PASSWORD, MYSQL_ALLOW_EMPTY_PASSWORD will install new mysql at startup' echo '=========mongod --help===========' echo `mongod --help` exit 0 fi DATADIR='/var/lib/mysql' BASEDIR='/usr/local/mysql' #-a:and -z:True if string is empty if [ "$1" = 'mysqld' ]; then # Get config #DATADIR="$("$@" --verbose --help --log-bin-index=`mktemp -u` 2>/dev/null | awk '$1 == "datadir" { print $2; exit }')" #if not exit /var/lib/mysql/mysql dir then run mysql_install_db # -a -z "$MYSQL_RANDOM_ROOT_PASSWORD" if [ ! -d "$DATADIR/mysql" ]; then if [ -z "$MYSQL_ROOT_PASSWORD" -a -z "$MYSQL_ALLOW_EMPTY_PASSWORD" ]; then echo >&2 'error: database is uninitialized and password option is not specified ' echo >&2 ' You need to specify one of MYSQL_ROOT_PASSWORD, MYSQL_ALLOW_EMPTY_PASSWORD' exit 1 fi mkdir -p "$DATADIR" chown -R mysql:mysql "$DATADIR" echo 'Initializing database' #下面的语句不要在末尾加2>&1 /dev/null可能会被识别为mysqld的参数,况且非bash可能不支持2>&1 mysql_install_db --user=mysql --datadir="$DATADIR" --rpm --basedir=$BASEDIR echo 'Database initialized' "$@" --skip-networking --basedir=$BASEDIR & pid="$!" mysql=( mysql --protocol=socket -uroot ) for i in {30..0}; do if echo 'SELECT 1' | "${mysql[@]}" &> /dev/null; then break fi echo 'MySQL init process in progress...' sleep 1 done if [ "$i" = 0 ]; then echo >&2 'MySQL init process failed.' exit 1 fi if [ -z "$MYSQL_INITDB_SKIP_TZINFO" ]; then # sed is for https://bugs.mysql.com/bug.php?id=20545 mysql_tzinfo_to_sql /usr/share/zoneinfo | sed 's/Local time zone must be set--see zic manual page/FCTY/' | "${mysql[@]}" mysql fi # if [ ! -z "$MYSQL_RANDOM_ROOT_PASSWORD" ]; then # MYSQL_ROOT_PASSWORD="$(pwgen -1 32)" # echo "GENERATED ROOT PASSWORD: $MYSQL_ROOT_PASSWORD" # fi "${mysql[@]}" <<-EOSQL -- What's done in this file shouldn't be replicated -- or products like mysql-fabric won't work SET @@SESSION.SQL_LOG_BIN=0; DELETE FROM mysql.user ; CREATE USER 'root'@'%' IDENTIFIED BY '${MYSQL_ROOT_PASSWORD}' ; GRANT ALL ON *.* TO 'root'@'%' WITH GRANT OPTION ; DROP DATABASE IF EXISTS test ; FLUSH PRIVILEGES ; EOSQL if [ ! -z "$MYSQL_ROOT_PASSWORD" ]; then mysql+=( -p"${MYSQL_ROOT_PASSWORD}" ) fi if [ "$MYSQL_DATABASE" ]; then echo "CREATE DATABASE IF NOT EXISTS \`$MYSQL_DATABASE\` ;" | "${mysql[@]}" mysql+=( "$MYSQL_DATABASE" ) fi if [ "$MYSQL_USER" -a "$MYSQL_PASSWORD" ]; then echo "CREATE USER '$MYSQL_USER'@'%' IDENTIFIED BY '$MYSQL_PASSWORD' ;" | "${mysql[@]}" if [ "$MYSQL_DATABASE" ]; then echo "GRANT ALL ON \`$MYSQL_DATABASE\`.* TO '$MYSQL_USER'@'%' ;" | "${mysql[@]}" fi echo 'FLUSH PRIVILEGES ;' | "${mysql[@]}" fi echo # for f in /docker-entrypoint-initdb.d/*; do # case "$f" in # *.sh) echo "$0: running $f"; . "$f" ;; # *.sql) echo "$0: running $f"; "${mysql[@]}" < "$f"; echo ;; # *.sql.gz) echo "$0: running $f"; gunzip -c "$f" | "${mysql[@]}"; echo ;; # *) echo "$0: ignoring $f" ;; # esac # echo # done if [ ! -z "$MYSQL_ONETIME_PASSWORD" ]; then echo >&2 echo >&2 'Sorry, this version of MySQL does not support "PASSWORD EXPIRE" (required for MYSQL_ONETIME_PASSWORD).' echo >&2 fi #kill the mysqld if ! kill -s TERM "$pid" || ! wait "$pid"; then echo >&2 'MySQL init process failed.' exit 1 fi echo echo 'MySQL init process done. Ready for start up.' echo fi chown -R mysql:mysql "$DATADIR" fi #re start the mysqld echo "formally start using cmd:$@" exec "$@"
构建镜像.etc
构建镜像
docker build -t db:1.0 .
删除已经退出的容器
docker rm -v `docker ps -aq`
在ubuntu低版本,如12.04(内核小于3.8)上安装docker的方法(低版本的docker使用上会有一些问题):
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys D8576A8BA88D21E9
sudo sh -c "echo deb http://mirror.yandex.ru/mirrors/docker/ docker main > /etc/apt/sources.list.d/docker.list"
sudo apt-get update
sudo apt-get install lxc-docker 可以加上--no-install-recommends避免安装git
总共安装了aufs-tools cgroup-lite liberror-perl lxc-docker lxc-docker-1.7.1
安装完docker的deb包后会自动创建docker组(groupadd docker)
并且启动docker,然后The UNIX socket /var/run/docker.sock is now readable and writable by members of the docker group.
可以将用户加入docker组来免去sudo权限:usermod -a -G docker user1 注销再登录就可以了(如果是ssh登录的,退出再重新登或者执行一次性的ssh shell命令就是登录后执行命令立刻退出)
镜像在节点之间的拷贝:
1.直接打包镜像docker save mynewimage > /tmp/mynewimage.tar
2.将容器保存为镜像再打包
docker commit 3a09b2588478 mynewimage
3.加载镜像 docker load < /tmp/mynewimage.tar
host主机上进行的设置
#docker容器中不能对内核的参数进行设置,是只读的,对于数据库需要调优的情况,需要在host上进行设置 #mongo echo never > /sys/kernel/mm/transparent_hugepage/enabled echo never > /sys/kernel/mm/transparent_hugepage/defrag #redis sysctl vm.overcommit_memory=1
run-image.sh启动镜像成为容器
#!/bin/bash # --cap-add=NET_ADMIN 特殊权限,容许设置网络 #-d 后台#--net='host' 使用host的网络配置,hostname,interface完全一样,不是隔离的了 #-it name:1.0 #容器运行起来后再启动数据库:docker exec mycontainer /path/to/script #first clear the exited containers,-v to remove associated volumns(really useful) #function for get this script's dir,used for locate relative resource files function get_script_dir () { SOURCE="${BASH_SOURCE[0]}" # While $SOURCE is a symlink, resolve it while [ -h "$SOURCE" ]; do DIR="$( cd -P "$( dirname "$SOURCE" )" && pwd )" SOURCE="$( readlink "$SOURCE" )" # If $SOURCE was a relative symlink (so no "/" as prefix, need to resolve it relative to the symlink base directory [[ $SOURCE != /* ]] && SOURCE="$DIR/$SOURCE" done DIR="$( cd -P "$( dirname "$SOURCE" )" && pwd )" return "$DIR" } param='master' if [ "$1" == "slave" ];then param='slave' elif [ -z "$1" ];then echo 'param: master/slave [bootstrap.sh] #last param specify if mount a external bootstrap.sh' exit 0 fi if [ ! -z "$2" ];then VOL_OPT="-v $2:/entrypoint.sh" fi #or you maybe account with name confict containers=`docker rm -v $(docker ps -a -q -f status=exited) 2> /dev/null` if [ ! -z "$containers" ];then echo 'removed exited container:' echo $containers fi #--net='host' #-t表示分配一个伪终端 #决定采用默认桥接方式,只需要对host进行端口映射,爬虫程序中的ip地址仍使用host的ip。 #如果要在单机上测试,仍需要配置网络 extra_param="" #--cap-add=NET_ADMIN PRE="$HOME/data" if [ $param == 'master' ];then NAME='zh' hostname='zh-master' names="--name $NAME -h $hostname" # extra_param=$extra_param" --net=host" # if set --net=host,the port mapping will be no use PORT='-p 16379:6379 -p 13306:3306 -p 37017:27017 -p 22122:22122' VOL="-v $PRE/mongo/:/data/mongo -v $PRE/redis:/data/redis -v $PRE/mysql/mysql/:/var/lib/mysql -v $PRE/mysql/conf.d/:/etc/mysql/conf.d -v $PRE/mysql/log/:/var/log/mysql -v $PRE/fdfs/:/data/fdfs" ENVs='-e MYSQL_ROOT_PASSWORD=f ' else NAME='zh-s' #hostname='zh-' names="--name $NAME" extra_param=$extra_param" --link zh:zh-m" PORT='-p 47017:27017 -p 23000:23000' #mongo's data area cannot be conflict with the master's setting VOL="-v $PRE/mongo.s/:/data/mongo -v $PRE/fdfs/:/data/fdfs" fi echo "container: $NAME starting.." CID=$(docker run $extra_param \ $names $ENVs $PORT $VOL $VOL_OPT \ -d -t zhihu:1.0 \ $param #192.168.1.10 #$1 $2 ) echo "id-> ${CID:0:5}" #可以通过-e MYSQL_PARAM=--character-set-server=utf8mb4 --xxx给mysql传递参数
容器退出后可直接使用下述命令启动:
docker start 名字 / ID
相关文章推荐
- Docker 制作Apache+PHP+GD+MongoDB+Redis镜像
- HBase vs. MongoDB vs. MySQL vs. Oracle vs. Redis,三大主流开源 NoSQL 数据库的 PK 两大主流传统 SQL 数据库
- HBase、MongoDB、MySQL、Oracle、Redis--nosql数据库与关系数据库对比
- 数据库高可用架构(MySQL、Oracle、MongoDB、Redis)
- 数据库高可用架构(MySQL、Oracle、MongoDB、Redis)
- python连接MySQL、MongoDB、Redis、memcache等数据库的方法
- 制作包含redis和mqtt的Docker镜像-支持多服务
- [转]数据库高可用架构(MySQL、Oracle、MongoDB、Redis)
- 数据库高可用架构(MySQL、Oracle、MongoDB、Redis)
- HBase vs. MongoDB vs. MySQL vs. Oracle vs. Redis,三大主流开源 NoSQL 数据库的 PK 两大主流传统 SQL 数据库
- HBase vs. MongoDB vs. MySQL vs. Oracle vs. Redis,三大主流开源 NoSQL 数据库的 PK 两大主流传统 SQL 数据库
- 数据库高可用架构(MySQL、Oracle、MongoDB、Redis)
- HBase vs. MongoDB vs. MySQL vs. Oracle vs. Redis,三大主流开源 NoSQL 数据库的 PK 两大主流传统 SQL 数据库
- Docker 制作mysql镜像
- memcached、redis、mysql、mongodb结合架构数据库系统
- python连接MySQL、MongoDB、Redis、memcache等数据库的方法
- 数据库高可用架构(MySQL、Oracle、MongoDB、Redis)
- 制作mysql的docker镜像
- docker 制作mysql镜像(一)
- 数据库高可用架构(MySQL、Oracle、MongoDB、Redis)