记一次Marathon被黑过程

前几天早上打开Mesos平台，发现有一个Killed的任务，MESOS_TASK_ID=hyakuhei.a318e232-28d9-11e6-bc8f-96bed1f124a2，名字很奇怪，并不是我跑的呀，然后去Marathon看一下，没有这个任务的container在运行，可能已经被删。这套Mesos + Marathon的平台在Bluemix的平台上跑的已经很稳定了，难道出现什么问题了？继续查看mesos日志，发现在两个slave节点跑过该任务，登录到slave docker ps -a一看，看到Image
Name吓一跳：

稍微懂点安全的人就知道Kali是什么系统，Metasploit是什么工具，赶紧查看last 记录，看看又没人登录成功的其他ip地址，查看系统的关键程序跟其他的节点对比都是正常的：

rpm -Vf /bin/ls

rpm -Vf /usr/sbin/sshd

rpm -Vf /sbin/ifconfig

rpm -Vf /usr/bin/docker 

zabbix看了一下当时的记录，出了I/O有点高之外，没有其他异常，我就好奇的docker start f9de5a11f30e看看里面是啥东西，try docker start 一下，发现启动报错（0ef6eeda359a是我自己启动的），那就去docker目录下面查查这个任务当时的配置信息吧：

被Mesos调度后的container信息都可以在此目录下查看到：





然后结合docker ps -a得到的name名称，被黑客用过的container名字目录是这个：

f9de5a11f30e4a88e186cdb443a72d0a8b66a8eb151a688d89a7ea9b8160df77，到目录下看到有个config.json，里面的内容如下：

{"State":{"Running":false,"Paused":false,"Restarting":false,"OOMKilled":false,"Dead":false,"Pid":0,"ExitCode":-1,"Error":"[8] System error: exec: \"chmod ug+rx 'tail -f /dev/null;' \u0026\u0026 exec 'tail -f /dev/null;' \": stat chmod ug+rx 'tail -f /dev/null;'
\u0026\u0026 exec 'tail -f /dev/null;' : no such file or directory","StartedAt":"0001-01-01T00:00:00Z","FinishedAt":"0001-01-01T00:00:00Z"},"ID":"f9de5a11f30e4a88e186cdb443a72d0a8b66a8eb151a688d89a7ea9b8160df77","Created":"2016-06-02T15:50:31.8048387Z","Path":"chmod
ug+rx 'tail -f /dev/null;' \u0026\u0026 exec 'tail -f /dev/null;' ","Args":[],"Config":{"Hostname":"bastion","Domainname":"shanker","User":"","AttachStdin":false,"AttachStdout":true,"AttachStderr":true,"PortSpecs":null,"ExposedPorts":null,"Tty":false,"OpenStdin":false,"StdinOnce":false,"Env":["LIBPROCESS_PORT=0","MESOS_AGENT_ENDPOINT=192.168.0.33:5051","MESOS_CHECKPOINT=1","MESOS_DIRECTORY=/tmp/mesos/slaves/025bd996-0430-46b3-afee-7d4b4248482b-S1/frameworks/d4bb23e8-a0b8-4dee-8500-27f663613ba0-0000/executors/marathon-hyakuhei.a318e232-28d9-11e6-bc8f-96bed1f124a2/runs/c8e9b672-ebda-4ade-98d5-4d7780f6686c","MESOS_EXECUTOR_ID=marathon-hyakuhei.a318e232-28d9-11e6-bc8f-96bed1f124a2","MESOS_EXECUTOR_SHUTDOWN_GRACE_PERIOD=5secs","MESOS_FRAMEWORK_ID=d4bb23e8-a0b8-4dee-8500-27f663613ba0-0000","MESOS_NATIVE_JAVA_LIBRARY=/usr/lib/libmesos-0.28.1.so","MESOS_NATIVE_LIBRARY=/usr/lib/libmesos-0.28.1.so","MESOS_RECOVERY_TIMEOUT=15mins","MESOS_SLAVE_ID=025bd996-0430-46b3-afee-7d4b4248482b-S1","MESOS_SLAVE_PID=slave(1)@192.168.0.33:5051","MESOS_SUBSCRIPTION_BACKOFF_MAX=2secs","MARATHON_APP_VERSION=2016-06-02T15:48:42.608Z","HOST=192.168.0.33","MARATHON_APP_RESOURCE_CPUS=1.0","PORT_10006=31027","MARATHON_APP_DOCKER_IMAGE=linuxkonsult/kali-metasploit","MESOS_TASK_ID=hyakuhei.a318e232-28d9-11e6-bc8f-96bed1f124a2","PORT=31027","MARATHON_APP_RESOURCE_MEM=128.0","PORTS=31027","MARATHON_APP_RESOURCE_DISK=0.0","MARATHON_APP_LABELS=","MARATHON_APP_ID=/hyakuhei","PORT0=31027","MESOS_SANDBOX=/mnt/mesos/sandbox","MESOS_CONTAINER_NAME=mesos-025bd996-0430-46b3-afee-7d4b4248482b-S1.c8e9b672-ebda-4ade-98d5-4d7780f6686c","PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin","DEBIAN_FRONTEND=noninteractive"],"Cmd":["chmod
ug+rx 'tail -f /dev/null;' \u0026\u0026 exec 'tail -f /dev/null;' "],"Image":"linuxkonsult/kali-metasploit","Volumes":null,"VolumeDriver":"","WorkingDir":"","Entrypoint":null,"NetworkDisabled":false,"MacAddress":"","OnBuild":null,"Labels":{}},"Image":"5284900a1876c960190c0d789f9562f285ac8231ecedbf533fa2371d1d6edc26","NetworkSettings":{"Bridge":"","EndpointID":"","Gateway":"","GlobalIPv6Address":"","GlobalIPv6PrefixLen":0,"HairpinMode":false,"IPAddress":"","IPPrefixLen":0,"IPv6Gateway":"","LinkLocalIPv6Address":"","LinkLocalIPv6PrefixLen":0,"MacAddress":"","NetworkID":"","PortMapping":null,"Ports":null,"SandboxKey":"","SecondaryIPAddresses":null,"SecondaryIPv6Addresses":null},"ResolvConfPath":"/var/lib/docker/containers/f9de5a11f30e4a88e186cdb443a72d0a8b66a8eb151a688d89a7ea9b8160df77/resolv.conf","HostnamePath":"/var/lib/docker/containers/f9de5a11f30e4a88e186cdb443a72d0a8b66a8eb151a688d89a7ea9b8160df77/hostname","HostsPath":"/var/lib/docker/containers/f9de5a11f30e4a88e186cdb443a72d0a8b66a8eb151a688d89a7ea9b8160df77/hosts","LogPath":"/var/lib/docker/containers/f9de5a11f30e4a88e186cdb443a72d0a8b66a8eb151a688d89a7ea9b8160df77/f9de5a11f30e4a88e186cdb443a72d0a8b66a8eb151a688d89a7ea9b8160df77-json.log","Name":"/mesos-025bd996-0430-46b3-afee-7d4b4248482b-S1.c8e9b672-ebda-4ade-98d5-4d7780f6686c","Driver":"devicemapper","ExecDriver":"native-0.2","MountLabel":"","ProcessLabel":"","RestartCount":0,"UpdateDns":false,"MountPoints":{"/mnt/mesos/sandbox":{"Name":"","Destination":"/mnt/mesos/sandbox","Driver":"","RW":true,"Source":"/tmp/mesos/slaves/025bd996-0430-46b3-afee-7d4b4248482b-S1/frameworks/d4bb23e8-a0b8-4dee-8500-27f663613ba0-0000/executors/marathon-hyakuhei.a318e232-28d9-11e6-bc8f-96bed1f124a2/runs/c8e9b672-ebda-4ade-98d5-4d7780f6686c","Relabel":""}},"Volumes":{"/mnt/mesos/sandbox":"/tmp/mesos/slaves/025bd996-0430-46b3-afee-7d4b4248482b-S1/frameworks/d4bb23e8-a0b8-4dee-8500-27f663613ba0-0000/executors/marathon-hyakuhei.a318e232-28d9-11e6-bc8f-96bed1f124a2/runs/c8e9b672-ebda-4ade-98d5-4d7780f6686c"},"VolumesRW":{"/mnt/mesos/sandbox":true},"AppArmorProfile":""}

看起来比较乱，仔细耐心看第一行就发现，其他这个实例是没有成功启动的，首先启动失败，之后黑客又重新启动，然后又失败了，就换到另外一个节点运行了一次，还是失败，所以这次没有造成很大损失，但是让我们看看他都执行了些啥命令，对做Docker安全也算是一次经验。

启动的实例名称是：hyakuhei

分配了1颗CPU,128M 内存，没有添加磁盘

端口映射的31027

使用的镜像是：linuxkonsult/kali-metasploit

执行的命令：Cmd":["chmod ug+rx 'tail -f /dev/null;' \u0026\u0026 exec 'tail -f /dev/null;' "

这一对奇怪的\u0026是被java编译后的结果，还原回去就是两个& 符号，所以命令就是

chmod ug+rx 'tailf -f /dev/null;' && exec 'tail -f /dev/null;'

看不出来有什么实际意义，估计某些命令也是被java编译进去了。

最后查看mesos的记录，也同样证明了我的判断，黑客是没有成功执行这项任务的：

现在想想好后怕，看来大家担心的Docker安全的问题确实是个Big Problem, 然后也怪我，Mesos 和Marathon启动的时候压根没加认证，查看了官方文档后，Marathon启动的时候加上 --http_credentials即可，然后Mesos启动时候加上--authenticate --credentials参数，让Mesos slave 连接到Master的时候加上认证。希望大家引以为鉴，不要讲Marathon暴漏到公网上，并且要加上认证，而且要不定时监控Docker运行情况。

欢迎补充！