VCS引起的oracle数据库异常重启一例
2017-01-03 15:33
344 查看
1. 环境描述
操作系统版本:SUSE Linux EnterpriseServer 10 sp2 (x86_64)
数据库版本:Oracle 11.1.0.7.16
VCS版本:5.1
blog地址:http://blog.csdn.net/hw_libo/article/details/41171561
2. 问题现象及分析
凌晨3:46左右,手机短信收到数据库异常告警。(1)查看数据库alert日志
[html] view
plain copy
print?
Sun Nov 16 03:46:51 2014
Stopping background process SMCO
Stopping background process FBDA
Shutting down instance: further logons disabled
Sun Nov 16 03:46:54 2014
kkjcre1p: unable to spawn jobq slave process, slot 0, error 1089
Stopping background process CJQ0
Stopping background process QMNC
Stopping background process MMNL
Stopping background process MMON
Sun Nov 16 03:47:03 2014
Shutting down instance (immediate)
License high water mark = 366
Sun Nov 16 03:51:52 2014
License high water mark = 366
USER (ospid: 8654): terminating the instance
Sun Nov 16 03:51:52 2014
ORA-1092 : opiodr aborting process unknown ospid (28656_47367034883680)
Sun Nov 16 03:51:52 2014
LGWR waiting for instance termination
并未发现较为明显的异常原因。
同时也查看了那个时间段的报告,发现数据库压力不大,也没有什么异常的SQL存在。
(2)检查OS资源使用
系统资源使用正常,IO也不高。
(3)查看messages日志
[html] view
plain copy
print?
Nov 16 03:42:46 boscodb AgentFramework[10894]: VCS ERROR V-16-2-13027 Thread(4149214112) Resource(ora_vg) - monitor procedure did not complete within the expected time.
Nov 16 03:42:46 boscodb Had[10258]: VCS ERROR V-16-2-13027 (boscodb) Resource(ora_vg) - monitor procedure did not complete within the expected time.
Nov 16 03:43:01 boscodb su: (to root) root on none
……
Nov 16 03:45:39 boscodb su: (to oracle) root on none
Nov 16 03:45:47 boscodb AgentFramework[10894]: VCS ERROR V-16-2-13210 Thread(4154813344) Agent is calling clean for resource(ora_vg) because 4 successive invocations of the monitor procedure did not complete within the expected time.
Nov 16 03:45:47 boscodb Had[10258]: VCS ERROR V-16-2-13210 (boscodb) Agent is calling clean for resource(ora_vg) because 4 successive invocations of the monitor procedure did not complete within the expected time.
……
Nov 16 03:46:48 boscodb AgentFramework[10894]: VCS ERROR V-16-2-13006 Thread(4153760672) Resource(ora_vg): clean procedure did not complete within the expected time.
Nov 16 03:46:48 boscodb Had[10258]: VCS ERROR V-16-2-13006 (boscodb) Resource(ora_vg): clean procedure did not complete within the expected time.
Nov 16 03:46:49 boscodb AgentFramework[10899]: VCS ERROR V-16-2-13067 Thread(4155194272) Agent is calling clean for resource(ora_archive) because the resource became OFFLINE unexpectedly, on its own.
Nov 16 03:46:50 boscodb Had[10258]: VCS ERROR V-16-2-13067 (boscodb) Agent is calling clean for resource(ora_archive) because the resource became OFFLINE unexpectedly, on its own.
Nov 16 03:46:51 boscodb AgentFramework[10899]: VCS ERROR V-16-2-13068 Thread(4155194272) Resource(ora_archive) - clean completed successfully.
原来是VCS出现了错误,监控vg超时了,这些错误信息在VCS的日志中也有记录。
3. 解决办法
检查了监控vg的moniter脚本,发现也没有异常,而且脚本在其他库,压力都比较大的都不曾出问题。当前的配置为:
[html] view
plain copy
print?
Application ora_vg (
StartProgram = "/opt/BOSCO_ora/bin/vg_start.sh"
StopProgram = "/opt/BOSCO_ora/bin/vg_stop.sh"
CleanProgram = "/opt/BOSCO_ora/bin/vg_clean.sh"
MonitorProgram = "/opt/BOSCO_ora/bin/vg_monitor.sh"
MonitorInterval = 30
MonitorTimeout = 30
)
决定暂时将监控时长加大到90s,可以使用两种方法:
(1)可以修改main.cf,但需要重启VCS,无操作时间窗口
(2)命令动态修改
haconf -makerw
hares -modify ora_vg MonitorTimeout 90 ## 调整监控时长为90s,默认是60s
haconf -dump
blog地址:http://blog.csdn.net/hw_libo/article/details/41171561
-- Bosco QQ:375612082
---- END ----
相关文章推荐
- VCS引起的oracle数据库异常重启一例
- VCS引起的oracle数据库异常重新启动一例
- 一例冷备份恢复引起的ORACLE数据库频繁重启故障
- linux系统中Oracle数据库异常关闭重启
- Oracle数据库索引失效,引起GoldenGate异常
- android 应用异常可以引起android系统崩溃重启
- Oracle数据库案例整理-Oracle系统运行时故障-RAC异常重启
- 解决因IP变更引起的oracle数据库连接异常“ORA-12541:TNS:无监听程序”
- 解决DocType引起AjaxTookit的ModulPopup显示异常问题的方法
- 导数据中的网络引起的异常
- 网闸引起的网络异常
- ZooKeeper配置引起Hbase和Kafka启动异常
- 【原】创建Hive表,分号分隔符“;”引起的异常
- Oracle数据库疑问:函数出现no_data_found不报异常
- 由多次使用Statement实例引起的Result set already closed异常的解决方案
- Excel打印质量引起的异常及解决方案
- 如何处理图片引起的OOM异常?
- 一个修饰符(static)引起的服务器异常
- windows下通过services.msc命令把某些服务停止引起系统运行异常的事宜