informatica 抛出ora-12592 :包错误 排查过程和解决办法

作者：lianghc

1.环境说明

informatica server 版本：9.6.1
informatica server上的oracle客户端：11.2.0.3
informatica server安装的操作系统：win server 2008 X64
oralce server :Oracle Database 11g Release 11.2.0.4.0 (oracle 环境是rac环境，有两个节点)
oralce server 安装在操作系统：redhat linux 6.4

2.问题描述

        最近在数据采集时总是出现ora-12592: TNS 包错误，出现的频率大概每周一次，这两天频繁出现，错误出现的情况随机不定，这一刻失败，下一刻重启又能成功，经过网上查询简单定位，此类错误我们初步定位为网络，防火墙的相关原因，咨询过客户网络管理员，管理员回复：服务器是内网连接，应该没有问题，服务期间没有防火墙。

3.informatica knowledge base

   在informatica 的社区找到一个kb-262530,描述如下，主要的操作是

添加客户端和服务器sqlnet.ora文件，并重新启动监听器和任何现有的工作/连接：
SQLNET.SEND_TIMEOUT=600

修改了相关的sqlnet后依然没有效果。

4.问题排查做出的尝试

在informatica 技术超群中资讯相关大牛，大多依然将问题定位为oracle 问题，其中有位说可能是informatica 服务端所依赖的oracle客户端有问题，我觉得说的有道理，于是将客户端换掉了，也有声音说是informatica的client（9.6.0）和server（9.6.1）的版本不一致造成的，我想笑的版本差异应该不是问题。
这时我的同事怀疑是session和process的问题，并做出如下解决：

1.查看当前连接进程数
SQL>selectcount(*) from v$process;
2.查看连接数上限
SQL>selectvalue from v$parameter where name = 'processes';

3.查看当前数据库的processes设置
SQL>show parameter processes
NAME                                TYPE        VALUE
db_writer_processes                 integer     1
gcs_server_processes                integer     0
job_queue_processes                 integer     10
log_archive_max_processes           integer     2
processes                           integer     150

SQL> show parameter sessions
NAME                                TYPE        VALUE
java_soft_sessionspace_limit        integer     0l
icense_max_sessions                integer     0
license_sessions_warning            integer     0
logmnr_max_persistent_sessions      integer     1
sessions                            integer     170
shared_server_sessions              integer

4.只要会话连接数超过上面的process数150或者sessions数170，再来一个的会话进程，就会产生12516错误。因此可以修改一下该值：
sessions=1.1*processes+5;//这个是sessions值和processes值的关系，最好按照这样做，
因此如果要将processes数设置为500，则sessions数必须为1.1*500+5=555

SQL>alter system set processes=500 scope=spfile;
系统已更改。
SQL> alter system set sessions=555 scope=spfile;
系统已更改。

5.重新启动数据库服务即可。
SQL>shutdown immediate
```
SQL>startup

5.INFA售后 给出的答复

努力尝试了很多方法后，问题依然没有进展，最后我们资讯了infa售后技术支持以及DBA

ORA-12592 错误是 Oracle 自身的 error message，是 TNS 软件在检测到不规范的包时发出的错误提示。

由于 InformaticaPowerExchange for Oracle Database 在抽取或装载数据到 Oracle 数据库时，采用的是 Oracle 的 Native driver（即原生驱动），通信的建立和控制管理，都是由Oracle自身的通信机制来完成的，所以，遇到这个问题，首先考虑到的，就是从 Oracle 客户端与服务器间的通信诊断开始。我在网上也查过，几乎所有与ora-12592 错误有关的网页，都认为问题出在底层网络上，或者 Oracle 通信参数的设定上。尽管，用户的管理员一再表示这是内网环境，防火墙也未启用，但鉴于这一明显的错误提示，如果条件允许，还是请客户的网络管理员来配合，联合诊断一下（比如开启 tracing，然后借用 Oracle 客户端通过网络往 Oracle 数据库服务器里进行 insert 操作，再分析该过程中记录下来的信息）。

6.DBA给出的答复

关于这个错误和网络关系比较大，如果服务器间没有防火墙，检查一下系统的防火墙是不是起来了。
关于这个错误：
错误12592意味着所提供的信息是不可接受的two-task层。它可以发生，如果数据本身是无效的或者包实际上是失败从Oracle Net（SQL * NET）层。
现在，如果数据包被损坏，Oracle网络层将报告错误发送信息到two-task层之前。错误本身发生在负责数据转换的数据库层中。这里是网络层后的层，是报告网络问题。这可能会出现一系列的原因：

1。查询本身会导致失败。
意思是不正确的查询。这可以被跟踪，如果转储发生重复，所有出现在相同类型的选择语句发生。

2。有一个进程失败（无论是客户端/服务器进程）。
如果应用程序挂在前端客户端上，并发送不完整的数据到
服务器进程。这是最难跟踪的，因为我们永远不知道在服务器端错误转储时客户端正在做什么。

3。在网络层有一个间歇性的通信损耗。
意思是一包信息没有传递。当发生较大的查询发生时，大多数情况发生，只有部分数据到达服务器进程。双任务尝试执行数据转换，但信息丢失。
这也是轨道非常困难因为唯一现实的选择是SQL * NET跟踪从的角度找出发生了什么的时候出现错误。我们看到底层的TCP或Winsock错误如果Windows客户参与。
如果是这样的话，那些可能会导致数据包传送的问题。

4 在某些情况下，客户端（可能是一个应用程序或ETL工作或进出口/ sqlloader）将间歇性失败与ora-12592而数据库报告任何错误或问题。
a 解决方案：应用补丁18841764可在不同平台上的11.2.0.4。
b 解决方法：添加以下行客户端和服务器sqlnet.ora文件，并重新启动监听器和任何现有的工作/连接：
SQLNET.SEND_TIMEOUT=600

7.最终解决问题

我们在DBA 的指导下，给数据库打了补丁18841764，在oracle的官网下载补丁：p18841764_112040_Linux-x86-64.zip

打补丁的指导文档：

Oracle Database 11g Release 11.2.0.4.0

ORACLE NET Patch for Bug# 18841764 for Linux-x86-64 Platforms

This patch is RAC Rolling Installable  - Please read My Oracle Support Document 244241.1 https://support.us.oracle.com/oip/faces/secure/km/DocumentDisplay.jspx?id=244241.1 Rolling Patch - OPatch Support for RAC.

This patch is non-Data Guard Standby-First Installable - Please read My Oracle Support Note 1265700.1 https://support.us.oracle.com/oip/faces/secure/km/DocumentDisplay.jspx?id=1265700.1 Oracle Patch Assurance - Data Guard Standby-First Patch Apply for details on how to remove risk and reduce downtime when applying this patch.

Released: Thu Jun 26 02:54:06 2014

This document describes how you can install the ORACLE NET interim patch for bug#  18841764 on your Oracle Database 11g Release 11.2.0.4.0

Note: This patch must be applied on the Oracle home of Oracle Database as well as Oracle Grid Infrastructure.

(I) Prerequisites
--------------------
Before you install or deinstall the patch, ensure that you meet the following requirements:

Note: In case of an Oracle RAC environment, meet these prerequisites on each of the nodes.

1.	Ensure that the Oracle home on which you are installing the patch or from which you are rolling back the patch is Oracle Database 11g Release 11.2.0.4.0.

2.      Oracle recommends that you use the latest version available for 11g Release 11.2.0.4.0. If you do not have  OPatch 11g Release 11.2.0.3.0 or the latest version available for 11g Release 11.2.0.4.0,then download it from patch# 6880880 for 11.2.0.4.0 release.

For information about OPatch documentation, including any known issues, see My Oracle Support Document 293369.1 OPatch documentation list: https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&id=224346.1 
3.	Ensure that you set (as the home user) the ORACLE_HOME environment variable to the Oracle home.

4.	Ensure that the $PATH definition has the following executables: make, ar, ld and nm. The location of these executables depends on your operating system. On many operating systems, they are located in /usr/ccs/bin.

5.	Ensure that you verify the Oracle Inventory because OPatch accesses it to install the patches. To verify the inventory, run the following command.
$ opatch lsinventory
Note:
-	If this command succeeds, it will list the Top-Level Oracle Products and one-off patches if any that are installed in the Oralce Home.
- Save the output so you have the status prior to the patch apply.
-	If the command displays some errors, then contact Oracle Support and resolve the issue first before proceeding further.

6.	(Only for Installation) Maintain a location for storing the contents of the patch ZIP file. In the rest of the document, this location (absolute path) is referred to as <PATCH_TOP_DIR>. Extract the contents of the patch ZIP file to the location (PATCH_TOP_DIR) you have created above. To do so, run the following command:
$ unzip -d <PATCH_TOP_DIR>  p18841764_112040_Linux-x86-64.zip

7.	(Only for Installation) Determine whether any currently installed interim patches conflict with this patch 18841764 as shown below:
$ cd <PATCH_TOP_DIR>/18841764
$ opatch prereq CheckConflictAgainstOHWithDetail -ph ./

The report will indicate the patches that conflict with this patch and the patches for which the current 18841764 is a superset.

Note:
When OPatch starts, it validates the patch and ensures that there are no conflicts with the software already installed in the ORACLE_HOME. OPatch categorizes conflicts into the following types:
-	Conflicts with a patch already applied to the ORACLE_HOME that is a subset of the patch you are trying to apply  - In this case, continue with the patch installation because the new patch contains all the fixes from the existing patch in the ORACLE_HOME. The subset patch will automatically be rolled back prior to the installation of the new patch.
-	Conflicts with a patch already applied to the ORACLE_HOME - In this case, stop the patch installation and contact Oracle Support Services.

8.	Ensure that you shut down all the services running from the Oracle home.
Note:
-	For a Non-RAC environment, shut down all the services running from the Oracle home.
-	For a RAC environment, shut down all the services (database, ASM, listeners, nodeapps, and CRS daemons) running from the Oracle home of the node you want to patch. After you patch this node, start the services on this node.Repeat this process for each of the other nodes of the Oracle RAC system. OPatch is used on only one node at a time.

9.      oracle executible has to be relinked and client shared library to be relinked

(II) Installation
-----------------
To install the patch, follow these steps:

1.	Set your current directory to the directory where the patch is located and then run the OPatch utility by entering the following commands:

$ cd <PATCH_TOP_DIR>/18841764

$ opatch apply

2.	Verify whether the patch has been successfully installed by running the following command:

$ opatch lsinventory

3.	Start the services from the Oracle home.

(III) Postinstallation
----------------------

none .

(IV) Deinstallation
----------------------
Ensure to follow the Prerequsites (Section I). To deinstall the patch, follow these steps:

1.	Deinstall the patch by running the following command:

$ opatch rollback -id 18841764

2.	Start the services from the Oracle home.

3.	Ensure that you verify the Oracle Inventory and compare the output with the one run before the patch installation and re-apply any patches that were rolled back as part of this patch apply. To verify the inventory, run the following command:

$ opatch lsinventory

(V)Postdeinstallation
--------------------------

After you deinstall the patch, reload the packages into the Oracle Database.
none .

(VI) Bugs Fixed by This Patch
---------------------------------
The following are the bugs fixed by this patch:
18841764: ORA-12592 TNS:BAD PACKET OR ORA-3137 OR ORA-1

--------------------------------------------------------------------------
Copyright 2014, Oracle and/or its affiliates. All rights reserved.
--------------------------------------------------------------------------

8.打完补丁遇到ora-12514错误

补丁 18841764 打完之后，数据库启动监听抛出ora-12514错误，在网上搜索都是说需要在Network/admin目录下找到listener.ora ，将标红添加进文件里（ORCL为监听的实例名）。

# listener.ora Network Configuration File: D:\..\network\admin\listener.ora

# Generated by Oracle configuration tools.

SID_LIST_LISTENER =

  (SID_LIST =

    (SID_DESC =

      (SID_NAME = PLSExtProc)

      (ORACLE_HOME = E:\..)

      (PROGRAM = extproc)

    )
   (SID_DESC =

    (GLOBAL_DBNAME = ORCL)

    (ORACLE_HOME = D:\..)

    (SID_NAME = ORCL)

    )

 )

LISTENER =

  (DESCRIPTION_LIST =

    (DESCRIPTION =

      (ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC1))

      (ADDRESS = (PROTOCOL = TCP)(HOST = onest)(PORT = 1521))

    )

  )

而实际上是因为打完补丁导致数据库存储的权限错误

9.修复数据库存储的权限错误

oracle 报错如下：

ALTER SYSTEM SET local_listener=' (ADDRESS=(PROTOCOL=TCP)(HOST=10.0.60.124)(PORT=1521))' SCOPE=MEMORY SID='oradc1';
NOTE: Loaded library: System
ORA-15025: could not open disk "/dev/raw/raw1"
ORA-27041: unable to open file
Linux-x86_64 Error: 13: Permission denied

---解决办法---
[root@datacenter1 ~]# ls -ltr /dev/raw/raw1
crw-rw---- 1 grid asmadmin 162, 1 Aug 11 15:05 /dev/raw/raw1

[grid@datacenter1 ~]$ cd /u01/product/grid/11.2.0/bin/
[grid@datacenter1 bin]$ ./setasmgidwrap o=/u01/product/oracle/11.2.0/db_1/bin/oracle