VCS引起的oracle数据库异常重启一例
2014-11-16 13:09
253 查看
1. 环境描述
操作系统版本:SUSE Linux Enterprise Server 10 sp2 (x86_64)数据库版本:Oracle 11.1.0.7.16
VCS版本:5.1
blog地址:/article/8468685.html
2. 问题现象及分析
凌晨3:46左右,手机短信收到数据库异常告警。(1)查看数据库alert日志
Sun Nov 16 03:46:51 2014 Stopping background process SMCO Stopping background process FBDA Shutting down instance: further logons disabled Sun Nov 16 03:46:54 2014 kkjcre1p: unable to spawn jobq slave process, slot 0, error 1089 Stopping background process CJQ0 Stopping background process QMNC Stopping background process MMNL Stopping background process MMON Sun Nov 16 03:47:03 2014 Shutting down instance (immediate) License high water mark = 366 Sun Nov 16 03:51:52 2014 License high water mark = 366 USER (ospid: 8654): terminating the instance Sun Nov 16 03:51:52 2014 ORA-1092 : opiodr aborting process unknown ospid (28656_47367034883680) Sun Nov 16 03:51:52 2014 LGWR waiting for instance termination
并未发现较为明显的异常原因。
同时也查看了那个时间段的报告,发现数据库压力不大,也没有什么异常的SQL存在。
(2)检查OS资源使用
系统资源使用正常,IO也不高。
(3)查看messages日志
Nov 16 03:42:46 boscodb AgentFramework[10894]: VCS ERROR V-16-2-13027 Thread(4149214112) Resource(ora_vg) - monitor procedure did not complete within the expected time. Nov 16 03:42:46 boscodb Had[10258]: VCS ERROR V-16-2-13027 (boscodb) Resource(ora_vg) - monitor procedure did not complete within the expected time. Nov 16 03:43:01 boscodb su: (to root) root on none …… Nov 16 03:45:39 boscodb su: (to oracle) root on none Nov 16 03:45:47 boscodb AgentFramework[10894]: VCS ERROR V-16-2-13210 Thread(4154813344) Agent is calling clean for resource(ora_vg) because 4 successive invocations of the monitor procedure did not complete within the expected time. Nov 16 03:45:47 boscodb Had[10258]: VCS ERROR V-16-2-13210 (boscodb) Agent is calling clean for resource(ora_vg) because 4 successive invocations of the monitor procedure did not complete within the expected time. …… Nov 16 03:46:48 boscodb AgentFramework[10894]: VCS ERROR V-16-2-13006 Thread(4153760672) Resource(ora_vg): clean procedure did not complete within the expected time. Nov 16 03:46:48 boscodb Had[10258]: VCS ERROR V-16-2-13006 (boscodb) Resource(ora_vg): clean procedure did not complete within the expected time. Nov 16 03:46:49 boscodb AgentFramework[10899]: VCS ERROR V-16-2-13067 Thread(4155194272) Agent is calling clean for resource(ora_archive) because the resource became OFFLINE unexpectedly, on its own. Nov 16 03:46:50 boscodb Had[10258]: VCS ERROR V-16-2-13067 (boscodb) Agent is calling clean for resource(ora_archive) because the resource became OFFLINE unexpectedly, on its own. Nov 16 03:46:51 boscodb AgentFramework[10899]: VCS ERROR V-16-2-13068 Thread(4155194272) Resource(ora_archive) - clean completed successfully.原来是VCS出现了错误,监控vg超时了,这些错误信息在VCS的日志中也有记录。
3. 解决办法
检查了监控vg的moniter脚本,发现也没有异常,而且脚本在其他库,压力都比较大的都不曾出问题。当前的配置为:
Application ora_vg ( StartProgram = "/opt/BOSCO_ora/bin/vg_start.sh" StopProgram = "/opt/BOSCO_ora/bin/vg_stop.sh" CleanProgram = "/opt/BOSCO_ora/bin/vg_clean.sh" MonitorProgram = "/opt/BOSCO_ora/bin/vg_monitor.sh" MonitorInterval = 30 MonitorTimeout = 30 )
决定暂时将监控时长加大到90s,可以使用两种方法:
(1)可以修改main.cf,但需要重启VCS,无操作时间窗口
(2)命令动态修改
haconf -makerw
hares -modify ora_vg MonitorTimeout 90 ## 调整监控时长为90s,默认是60s
haconf -dump
blog地址:/article/8468685.html
-- Bosco QQ:375612082
---- END ----
-------------------------------------------------------------------------------------------------------
版权所有,文章允许转载,但必须以链接方式注明源地址,否则追究法律责任!
相关文章推荐
- VCS引起的oracle数据库异常重启一例
- VCS引起的oracle数据库异常重新启动一例
- 一例冷备份恢复引起的ORACLE数据库频繁重启故障
- android 应用异常可以引起android系统崩溃重启
- Oracle数据库案例整理-Oracle系统运行时故障-RAC异常重启
- 解决因IP变更引起的oracle数据库连接异常“ORA-12541:TNS:无监听程序”
- linux系统中Oracle数据库异常关闭重启
- Oracle数据库索引失效,引起GoldenGate异常
- 如何设置当程序出现异常后进行重启
- AVR中断引起重启
- 在linux下python爬虫进程发生异常时自动重启直至正常结束的方法
- (oracle数据库基础)第六章 游标和异常管理
- 异常关闭MyEclipse 8.6后,不能重启
- Android学习 - Service被关闭后自动重启,解决被异常kill 服务
- 【转】Android 中处理崩溃异常并重启程序出现页面重叠的问题
- 【转载】通用 application 彻底退出应用 获崩溃异常,保存错误日志,并重启应用
- Oracle数据库归档日志满后造成系统宕机解决一例
- MOM自定义规则一例:监控服务器重启事件 推荐
- 网站出现问题Linux重启oracle数据库的三大技巧
- 键盘勾子匿名方法引起的异常