服务器宕机记录16.12.14

阿里云主机
16.12.14日11：37分宕机，第一时间收到短信，ssh远程远法登录，阿里云控制台，无法登录。查看CPU和内存，没有鸟用，只能重启。
11：40分，正常登录主机，查看messages
Dec 14 11:30:50 iZ258fuhh93Z systemd: Started Session 178481 of user root.
Dec 14 11:30:50 iZ258fuhh93Z systemd: Starting Session 178481 of user root.
Dec 14 11:30:51 iZ258fuhh93Z systemd: Started Session 178479 of user deploy.
Dec 14 11:30:51 iZ258fuhh93Z systemd: Starting Session 178479 of user deploy.
Dec 14 11:30:52 iZ258fuhh93Z systemd: Started Session 178480 of user deploy.
Dec 14 11:30:57 iZ258fuhh93Z systemd: Starting Session 178480 of user deploy.
Dec 14 11:33:41 iZ258fuhh93Z systemd-logind: Failed to start session scope session-178482.scope: Connection timed out (null)
Dec 14 11:34:20 iZ258fuhh93Z systemd-logind: Failed to abandon session scope: Connection timed out
Dec 14 11:35:37 iZ258fuhh93Z crond: sendmail: fatal: parameter inet_interfaces: no local interface found for ::1
Dec 14 11:35:37 iZ258fuhh93Z systemd: Started Session 178485 of user deploy.
Dec 14 11:35:37 iZ258fuhh93Z systemd: Starting Session 178485 of user deploy.
Dec 14 11:35:37 iZ258fuhh93Z systemd: Failed to start Session 178482 of user deploy.
Dec 14 11:35:37 iZ258fuhh93Z systemd: Failed to start Session 178483 of user deploy.
Dec 14 11:35:37 iZ258fuhh93Z systemd: Started Session 178484 of user deploy.
Dec 14 11:35:37 iZ258fuhh93Z systemd: Starting Session 178484 of user deploy.
Dec 14 11:35:37 iZ258fuhh93Z systemd: Started Session 178486 of user deploy.
Dec 14 11:35:37 iZ258fuhh93Z systemd: Starting Session 178486 of user deploy.
Dec 14 11:36:35 iZ258fuhh93Z systemd-logind: Failed to abandon session scope: Connection timed out
Dec 14 11:36:59 iZ258fuhh93Z systemd: Started Session 178487 of user deploy.
Dec 14 11:36:59 iZ258fuhh93Z systemd: Starting Session 178487 of user deploy.
Dec 14 11:36:59 iZ258fuhh93Z crond: sendmail: fatal: parameter inet_interfaces: no local interface found for ::1
Dec 14 11:37:06 iZ258fuhh93Z crond: sendmail: fatal: parameter inet_interfaces: no local interface found for ::1
Dec 14 11:37:07 iZ258fuhh93Z systemd: Started Session 178488 of user deploy.
Dec 14 11:37:07 iZ258fuhh93Z systemd: Starting Session 178488 of user deploy.
Dec 14 11:37:13 iZ258fuhh93Z crond: sendmail: fatal: parameter inet_interfaces: no local interface found for ::1
Dec 14 11:37:13 iZ258fuhh93Z crond: sendmail: fatal: parameter inet_interfaces: no local interface found for ::1
Dec 14 11:39:34 iZ258fuhh93Z kernel: atkbd serio0: Unknown key released (translated set 2, code 0x0 on isa0060/serio0).
Dec 14 11:39:34 iZ258fuhh93Z kernel: atkbd serio0: Use 'setkeycodes 00 <keycode>' to make it known.
Dec 14 11:39:34 iZ258fuhh93Z kernel: atkbd serio0: Unknown key released (translated set 2, code 0x0 on isa0060/serio0).
Dec 14 11:39:34 iZ258fuhh93Z kernel: atkbd serio0: Use 'setkeycodes 00 <keycode>' to make it known.
Dec 14 11:39:34 iZ258fuhh93Z systemd: getty@tty1.service has no holdoff time, scheduling restart.
Dec 14 11:39:35 iZ258fuhh93Z systemd: Failed to fork: Cannot allocate memory
Dec 14 11:39:35 iZ258fuhh93Z systemd: getty@tty2.service: Supervising process 48 which is not our child. We'll most likely not notice when it exits.

Dec 14 11:39:39 iZ258fuhh93Z systemd: Started Session 178490 of user deploy.
Dec 14 11:39:40 iZ258fuhh93Z systemd: Starting Session 178490 of user deploy.
Dec 14 11:39:40 iZ258fuhh93Z systemd: Started Session 178489 of user deploy.
Dec 14 11:39:40 iZ258fuhh93Z systemd: Starting Session 178489 of user deploy.
Dec 14 11:40:53 iZ258fuhh93Z rsyslogd: [origin software="rsyslogd" swVersion="7.4.7" x-pid="471" x-info="http://www.rsyslog.com"] star
Dec 14 19:40:46 iZ258fuhh93Z journal: Runtime journal is using 8.0M (max allowed 189.5M, trying to leave 284.3M free of 1.8G available → current limit 189.5M).
Dec 14 19:40:46 iZ258fuhh93Z kernel: Initializing cgroup subsys cpuset
Dec 14 19:40:46 iZ258fuhh93Z kernel: Initializing cgroup subsys cpu
Dec 14 19:40:46 iZ258fuhh93Z kernel: Initializing cgroup subsys cpuacct
>>>>>>> 重启

Dec 14 19:40:46 iZ258fuhh93Z kernel: Linux version 3.10.0-327.10.1.el7.x86_64 (builder@kbuilder.dev.centos.org) (gcc version 4.8.3 20140911 (Red Hat 4.8.3-9) (GCC) ) #1 SMP
Tue Feb 16 17:03:50 UTC 2016
Dec 14 19:40:46 iZ258fuhh93Z kernel: Command line: BOOT_IMAGE=/boot/vmlinuz-3.10.0-327.10.1.el7.x86_64 root=UUID=6634633e-001d-43ba-8fab-202f1df93339 ro crashkernel=auto vco
nsole.font=latarcyrheb-sun16 vconsole.keymap=us biosdevname=0 rhgb quiet LANG=en_US.UTF-8 net.ifnames=0
Dec 14 19:40:46 iZ258fuhh93Z kernel: e820: BIOS-provided physical RAM map:
Dec 14 19:40:46 iZ258fuhh93Z kernel: BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
Dec 14 19:40:46 iZ258fuhh93Z kernel: BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
Dec 14 19:40:46 iZ258fuhh93Z kernel: BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved

分析日志发现主要问题是内存耗尽，无法创建ssh链接进程，可想，可能是什么服务耗尽了内存，导致服务器宕机
该问题很有可能和昨天晚上上的新任务有关系，要具体业务进行分析。

其次发现每次的cron任务里有发送邮件失败的问题，上网找到解决方案，解决它：
sendmail: fatal: parameter inet_interfaces: no local interface found for ::1原因跟IPv6有关，解决办法就是让postfix只用IPv4。打开文件：/etc/postfix/main.cf把inet_protocols = all改成inet_protocols = ipv4完了之后执行service postfix restart