OPatch failed with error code 41 Following executables are active : /lib/libclntsh.so.11.1

一. 问题描述：

[oracle@erpdb01 16817656]$
/u01/PROD/db/tech_st/11.2.0/OPatch/opatch apply

Oracle Interim Patch Installer version 11.2.0.3.10

Copyright (c) 2016, Oracle Corporation. All rights reserved.

Oracle Home : /u01/PROD/db/tech_st/11.2.0

Central Inventory : /u01/PSPROD/oraInventory

from : /u01/PROD/db/tech_st/11.2.0/oraInst.loc

OPatch version : 11.2.0.3.10

OUI version : 11.2.0.4.0

Log file location : /u01/PROD/db/tech_st/11.2.0/cfgtoollogs/opatch/16817656_Jun_29_2016_20_46_34/apply2016-06-29_20-46-34PM_1.log

Applying interim patch '16817656' to OH '/u01/PROD/db/tech_st/11.2.0'

Verifying environment and performing prerequisite checks...

Prerequisite check "CheckActiveFilesAndExecutables" failed.

The details are:

Following executables are active :

/u01/PROD/db/tech_st/11.2.0/lib/libclntsh.so.11.1

Log file location: /u01/PROD/db/tech_st/11.2.0/cfgtoollogs/opatch/16817656_Jun_29_2016_20_46_34/apply2016-06-29_20-46-34PM_1.log

Recommended actions: OPatch needs to modify files which are being used by some processes.

OPatch failed with error code 41

[oracle@erpdb01 16817656]$

二.问题分析：

根据报错提示显示活动的进程libclntsh.so.11.1阻止了opath的进行
查看此进程在干嘛
[root@erpdb01 ~]# lsof /u01/PROD/db/tech_st/11.2.0/lib/libclntsh.so.11.1

COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME

java 84656 oracle mem REG 252,3 53812404 4669464 /u01/PROD/db/tech_st/11.2.0/lib/libclntsh.so.11.1

emagent 84684 oracle mem REG 252,3 53812404 4669464 /u01/PROD/db/tech_st/11.2.0/lib/libclntsh.so.11.1

三.问题解决：

停掉em12c的进程
[oracle@erpdb01 ~]$ cd /u01/em12c/agent_inst/bin/

[oracle@erpdb01 bin]$ ./emctl stop agent

Oracle Enterprise Manager Cloud Control 12c Release 3

Copyright (c) 1996, 2013 Oracle Corporation. All rights reserved.

Stopping agent ..... stopped.

[oracle@erpdb01 bin]$

查看阻止进程状态：
[root@erpdb01 ~]# lsof /u01/PROD/db/tech_st/11.2.0/lib/libclntsh.so.11.1

COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME

java 84656 oracle mem REG 252,3 53812404 4669464 /u01/PROD/db/tech_st/11.2.0/lib/libclntsh.so.11.1

emagent 84684 oracle mem REG 252,3 53812404 4669464 /u01/PROD/db/tech_st/11.2.0/lib/libclntsh.so.11.1

[root@erpdb01 ~]#

阻止进程还在杀掉em进程
[root@erpdb01 ~]# kill -9 84684

[root@erpdb01 ~]# lsof /u01/PROD/db/tech_st/11.2.0/lib/libclntsh.so.11.1

COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME

java 84656 oracle mem REG 252,3 53812404 4669464 /u01/PROD/db/tech_st/11.2.0/lib/libclntsh.so.11.1

[root@erpdb01 ~]#

再次apply：
[oracle@erpdb01 16817656]$ /u01/PROD/db/tech_st/11.2.0/OPatch/opatch apply

Oracle Interim Patch Installer version 11.2.0.3.10

Copyright (c) 2016, Oracle Corporation. All rights reserved.

Oracle Home : /u01/PROD/db/tech_st/11.2.0

Central Inventory : /u01/PSPROD/oraInventory

from : /u01/PROD/db/tech_st/11.2.0/oraInst.loc

OPatch version : 11.2.0.3.10

OUI version : 11.2.0.4.0

Log file location : /u01/PROD/db/tech_st/11.2.0/cfgtoollogs/opatch/16817656_Jun_29_2016_20_58_42/apply2016-06-29_20-58-42PM_1.log

Applying interim patch '16817656' to OH '/u01/PROD/db/tech_st/11.2.0'

Verifying environment and performing prerequisite checks...

All checks passed.

Provide your email address to be informed of security issues, install and

initiate Oracle Configuration Manager. Easier for you if you use your My

Oracle Support Email address/User Name.

Visit http://www.oracle.com/support/policies.html for details.

Email address/User Name:

You have not provided an email address for notification of security issues.

Do you wish to remain uninformed of security issues ([Y]es,
o)
: y

This node is part of an Oracle Real Application Cluster.

Remote nodes: 'erpdb02'

Local node: 'erpdb01'

Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.

(Oracle Home = '/u01/PROD/db/tech_st/11.2.0')

Is the local system ready for patching? [y|n]

y

User Responded with: Y

Backing up files...

Patching component oracle.network.rsf, 11.2.0.4.0...

Patching component oracle.rdbms.rsf, 11.2.0.4.0..

Patching component oracle.rdbms, 11.2.0.4.0...

Verifying the update...

The local system has been patched. You can restart Oracle instances on it.

Patching in rolling mode.

The node 'erpdb02' will be patched next.

Please shutdown Oracle instances running out of this ORACLE_HOME on 'erpdb02'.

(Oracle Home = '/u01/PROD/db/tech_st/11.2.0')

Is the node ready for patching? [y|n]

n

User Responded with: N

ApplySession exits on request

You may exit the patching session and patch remaining nodes later from an un-patched node. Do you want to continue?

Do you want to proceed? [y|n]

n

User Responded with: N

Patching aborted on node "erpdb02", no remaining nodes.

OPatch failed to propagate the patch to the remote node(s).

Run the patch again on each of the remote node(s) using -local option.

Details: Patching aborted on node "erpdb02", no remaining nodes.

OPatch failed: OUI-67081:OPatch failed to propagate the patch to the remote node(s).

Run the patch again on each of the remote node(s) using -local option.

Details: Patching aborted on node "erpdb02", no remaining nodes.

Log file location: /u01/PROD/db/tech_st/11.2.0/cfgtoollogs/opatch/16817656_Jun_29_2016_20_58_42/apply2016-06-29_20-58-42PM_1.log

OPatch failed with error code 202

[oracle@erpdb01 16817656]$

检查补丁状态：
[oracle@erpdb01 ~]$ /u01/PROD/db/tech_st/11.2.0/OPatch/opatch lsinventory

Oracle Interim Patch Installer version 11.2.0.3.10

Copyright (c) 2016, Oracle Corporation. All rights reserved.

Oracle Home : /u01/PROD/db/tech_st/11.2.0

Central Inventory : /u01/PSPROD/oraInventory

from : /u01/PROD/db/tech_st/11.2.0/oraInst.loc

OPatch version : 11.2.0.3.10

OUI version : 11.2.0.4.0

Log file location : /u01/PROD/db/tech_st/11.2.0/cfgtoollogs/opatch/opatch2016-06-29_21-00-55PM_1.log

Lsinventory Output file location : /u01/PROD/db/tech_st/11.2.0/cfgtoollogs/opatch/lsinv/lsinventory2016-06-29_21-00-55PM.txt

--------------------------------------------------------------------------------

Local Machine Information::

Hostname: erpdb01.szgas.com

ARU platform id: 226

ARU platform description:: Linux x86-64

Installed Top-level Products (2):

Oracle Database 11g 11.2.0.4.0

Oracle Database 11g Examples 11.2.0.4.0

There are 2 products installed in this Oracle Home.

Interim patches (5) :

Patch 16817656 : applied on Wed Jun 29 20:59:23 CST 2016

Unique Patch ID: 18867694

Created on 29 Apr 2015, 15:37:22 hrs PST8PDT

Bugs fixed:

16817656

Patch 19852360 : applied on Sat Jun 06 02:07:09 CST 2015

Unique Patch ID: 18170553

Created on 20 Oct 2014, 08:17:43 hrs PST8PDT

Bugs fixed:

19852360

Patch 20420937 : applied on Sat Jun 06 02:05:13 CST 2015

Unique Patch ID: 18573450

Patch description: "OCW Patch set update : 11.2.0.4.6 (20420937)"

Created on 27 Mar 2015, 15:19:23 hrs PST8PDT

Bugs fixed:

18328800, 19270660, 18691572, 20365005, 17750548, 17387214, 17617807

14497275, 17733927, 18180541, 18962892, 17292250, 17378618, 16759171

17065496, 13991403, 17273020, 17155238, 18053580, 18261183, 17039197

16317771, 17947785, 10052729, 20340620, 18199185, 18399991, 18024089

20246071, 18414137, 17001914, 17927970, 14378120, 16346413, 15986647

18068871, 18143836, 16206997, 19168690, 20235511, 18343490, 16613232

19276791, 17722664, 12928658, 16249829, 18520351, 18226143, 18265482

18229842, 17172091, 17818075, 18231837, 14373486, 17483479, 18120545

18729166, 13843841, 17405302, 18709496, 18330979, 18187697, 14525998

20531190, 14385860, 19479503, 18348155, 18370031, 17516024, 17764053

17551223, 14671408, 14207615, 18272135, 17500165, 18875012, 19558324

18464784, 18848125, 19241857, 17955615, 14693336, 20315294, 16284825

20014326, 17352230, 17238586, 17089344, 17405605, 17531342, 17640316

17159489, 16543190, 18346135, 16281493, 17481314, 15986311, 17208793

18700935, 18999857, 14076173, 18428146, 17435488, 18352845, 18352846

17387779, 17391726, 20141091, 15832129, 17305100, 19885321, 16901346

17985714, 18536826, 17780903, 18752378, 18946768, 16875342, 16876500

19955755, 16429265, 18336452, 17273003, 17059927, 17046460, 18053631

16867761, 20235486, 15869775, 19642566, 17447588, 15920201

Patch 20449729 : applied on Sat Jun 06 02:00:58 CST 2015

Unique Patch ID: 18624316

Patch description: "DATABASE PATCH FOR EXADATA (APR2015 - 11.2.0.4.16) : (20449729)"

Created on 25 Mar 2015, 03:47:42 hrs PST8PDT

Sub-patch 20059375; "DATABASE PATCH FOR EXADATA (JAN2015 - 11.2.0.4.15) : (20059375)"

Sub-patch 19893780; "DATABASE PATCH FOR EXADATA (Dec2014 - 11.2.0.4.14) : (19893780)"

Sub-patch 19698715; "DATABASE PATCH FOR EXADATA (Nov2014 - 11.2.0.4.13) : (19698715)"

Sub-patch 19495869; "DATABASE PATCH FOR EXADATA (OCT2014 - 11.2.0.4.12) : (19495869)"

Sub-patch 19307276; "DATABASE PATCH FOR EXADATA (SEP2014 - 11.2.0.4.11) : (19307276)"

Sub-patch 19061859; "DATABASE PATCH FOR EXADATA (AUG2014 - 11.2.0.4.10) : (19061859)"

Sub-patch 18825509; "DATABASE PATCH FOR EXADATA (JUL2014 - 11.2.0.4.9) : (18825509)"

Sub-patch 18642122; "DATABASE PATCH FOR EXADATA (JUN2014 - 11.2.0.4.8) : (18642122)"

Sub-patch 18552960; "DATABASE PATCH FOR EXADATA (MAY2014 - 11.2.0.4.7) : (18552960)"

Sub-patch 18293775; "DATABASE PATCH FOR EXADATA (APR2014 - 11.2.0.4.6) : (18293775)"

Sub-patch 18136151; "DATABASE PATCH FOR EXADATA (MAR2014 - 11.2.0.4.5) : (18136151)"

Sub-patch 18006299; "DATABASE PATCH FOR EXADATA (FEB2014 - 11.2.0.4.4) : (18006299)"

Sub-patch 17943261; "DATABASE PATCH FOR EXADATA (JAN2014 - 11.2.0.4.3) : (17943261)"

Sub-patch 17741631; "DATABASE PATCH FOR EXADATA (DEC 2013 - 11.2.0.4.2) : (17741631)"

Sub-patch 17628006; "DATABASE PATCH FOR EXADATA (NOV 2013 - 11.2.0.4.1) : (17628006)"

Bugs fixed:

17288409, 16188701, 16930924, 18607546, 17205719, 13640676, 17811429

20506699, 17816865, 17922254, 17754782, 13364795, 16934803, 17311728

18418934, 16809786, 17284817, 17441661, 14275161, 18685209, 16477664

18255105, 16992075, 14193240, 17446237, 14015842, 18324129, 19972569

18317132, 18528020, 18029594, 17375354, 17449815, 17208934, 19463897

17265093, 13866822, 17235750, 17982555, 17478514, 18317531, 14338435

18235390, 13944971, 20142975, 17079301, 17811789, 19524158, 16929165

18704244, 17031322, 20506706, 17546973, 14054676, 17088068, 19168442

16885125, 18264060, 17346091, 18780342, 17343514, 19680952, 18471685

19240264, 18819257, 17901041, 19211724, 16199543, 17775506, 13951456

16315398, 18744139, 16850630, 18863094, 18840932, 17561405, 18767554

19049453, 18673304, 17883081, 14255128, 19188927, 19915271, 18641419

17284345, 18262334, 17006183, 18277454, 16833527, 17249711, 10136473

16015637, 18051556, 17865671, 17852463, 18773823, 18554871, 17853498

16988491, 17635021, 18334586, 19220183, 19487147, 19230722, 18966843

19617921, 17551709, 18879319, 17588480, 19827973, 17344412, 17842825

18828868, 17025461, 13609098, 17401353, 11883252, 17239687, 17602269

19197175, 18316692, 17313525, 12611721, 19544839, 18025431, 18964939

17600719, 18191164, 17571306, 19393542, 18482502, 19466309, 18419770

17040527, 17165204, 18098207, 16785708, 17564992, 17465741, 16180763

17174582, 16777840, 12982566, 19463893, 16875449, 12816846, 17237521

19788303, 19358317, 18665986, 17811438, 17811447, 17945983, 18762750

17080195, 16912439, 17184721, 14373152, 17540469, 18061914, 17282229

13512939, 16667538, 18331850, 18125929, 18202441, 17082359, 18723434

16299727, 19554106, 14034426, 18339044, 19458377, 17752995, 17209410

17785870, 17254374, 17891943, 17767676, 18077682, 17258090, 16668584

12608451, 17205005, 18384391, 17040764, 17381384, 15913355, 18356166

14084247, 20506715, 13853126, 18663279, 18203837, 14245531, 16043574

17848897, 17877323, 17402822, 14486653, 17468141, 14313306, 17786518

17912217, 17037130, 18155762, 16956380, 17478145, 17394950, 18641461

18189036, 18619917, 17027426, 20490588, 16268425, 19257578, 18436307

17265217, 13498382, 17634921, 17734862, 17443671, 18000422, 16804061

18783969, 14263190, 17571039, 18798414, 19014915, 18497527, 16344544

18247351, 18025214, 18009564, 17621643, 13073613, 19527817, 17591148

14354737, 17781859, 14764840, 18135678, 19049932, 18614015, 17853355

18134680, 18362222, 18192858, 17082612, 16472716, 17835048, 18830412

17050888, 17936109, 14010183, 17325413, 16466530, 17932831, 18747196

17799428, 18832544, 18284357, 17761775, 16721594, 17082983, 19689469

18148383, 19402853, 18532136, 17302277, 18084625, 15990359, 19488514

18203835, 13639209, 18499306, 17297939, 16731148, 19296555, 17811456

18205490, 13829543, 14133975, 17215560, 19186845, 17694209, 18091059

17385178, 16830594, 8322815, 17586955, 17670355, 17201159, 17655634

18331812, 19730508, 18868646, 17648596, 16220077, 16069901, 18485835

17393915, 17348614, 17274537, 17957017, 18096714, 17308789, 18436647

17806696, 14285317, 19289642, 18284763, 14764829, 17622427, 18328509

18169273, 16943711, 14368995, 17346671, 18996843, 17783588, 16618694

15902104, 17672719, 18856999, 18783224, 17851160, 19540573, 17546761

17798953, 19624199, 18273830, 17501296, 19972566, 17726838, 16384983

20217173, 15834345, 19410056, 20299013, 17360606, 13645875, 19181525

18199537, 16542886, 17889549, 18938517, 14565184, 17071721, 20299015

17610798, 17397545, 18230522, 18221857, 16360112, 19769489, 12905058

18961101, 18889295, 18641451, 19348649, 12747740, 18430495, 18867023

19291380, 17230905, 18851298, 18317074, 17016369, 17042658, 17314427

14602788, 18686405, 19972568, 16825679, 17158214, 19788842, 18508861

18508164, 19546825, 14657740, 17332800, 13837378, 18483595, 17186905

19972564, 18315328, 17437634, 19006849, 17836820, 19013183, 17296856

18674024, 14333054, 17232014, 16855292, 17801017, 17762296, 14692762

17705023, 17179434, 17997507, 19121551, 18331944, 19854503, 17469624

19309466, 19146626, 18304997, 18681862, 18554763, 17390160, 18456514

13955826, 16306373, 18139690, 17501491, 17752121, 17993995, 17299889

18940497, 17889583, 18673325, 19031341, 18818847, 18293054, 17242746

17452841, 19587324, 17951233, 17649265, 18094246, 19615136, 17011832

16820228, 16870214, 17477958, 18522509, 16091637, 16971958, 17323222

16595641, 16524926, 18228645, 16980342, 18282562, 14822091, 17596908

13982287, 17156148, 18031668, 16494615, 19023822, 18077632, 18760484

17545847, 18304693, 17614134, 13558557, 17341326, 17891946, 17716305

18133214, 18074623, 17496884, 16392068, 19271443, 18092127, 17736165

18554519, 18280356, 18365267, 17614227, 18440047, 19403858, 16903536

14106803, 18973907, 18673342, 16359751, 17389192, 17612828, 16194160

17006570, 17721717, 17390431, 17570240, 18417036, 14254610, 18456874

16863422, 18325460, 19727057, 14460384, 18226122, 16422541, 17267114

19972570, 18244962, 18203838, 18765602, 16198143, 17246576, 14829250

17835627, 12743640, 14000767, 18247991, 17839474, 14458214, 18684802

12716670, 16618055, 16692232, 17786278, 17227277, 16042673, 16314254

19308965, 17952061, 16228604, 18154779, 18061271, 18689530, 16837842

17393683, 13498243, 18730542, 17787259, 20074391, 15861775, 19900800

19394032, 16399083, 18191542, 14468157, 18018515, 18508266, 18433189

20543011, 18416368, 18701328, 18260550, 17080436, 16613964, 18604144

17036973, 16898135, 16579084, 18803246, 18384537, 18280813, 20296213

16901385, 20241212, 15979965, 13651346, 13770500, 19174639, 18441944

16450169, 17357979, 9756271, 14046443, 18160822, 17892268, 11733603

16285691, 17587063, 18180390, 16538760, 18193833, 12578873, 18665660

13816053, 17238511, 16271658, 17824637, 13877071, 14309884, 17277476

16757715, 17328159, 16571443, 18306996, 14852021, 17853456, 18674047

12364061, 17799716

Patch 4247037 : applied on Thu Nov 28 16:51:39 CST 2013

Unique Patch ID: 16751181

Created on 03 SEP 2013, 09:52:00 hrs US/Pacific

Bugs fixed:

4247037

Rac system comprising of multiple nodes

Local node = erpdb01

Remote node = erpdb02

--------------------------------------------------------------------------------

OPatch succeeded.

[oracle@erpdb01 ~]$