Monitoring the Cluster in Real Time with CHM (Cluster Health Monitor)

http://levipereira.wordpress.com/2011/07/19/monitoring-the-cluster-in-real-time-with-chm-cluster-health-monitor/

WhyClusterHealthMonitor?

OracleClusterware&Oracledatabaseperformance/noderebootduetolackofCPU/MemoryresourcescauseCustomerstoaskhowtomonitortheirOS.Somecustomershaverudimentaryscriptsthatutilizevmstat,mpstatbuttheyareoftennodecollected
atregularintervals.Insomecases,wehaveseencustomerscollectthisonceperhourwhichdoesnotmakeitveryusefulwhenthenodeishung/evitedviarebootinthemiddleofthehour.OSwatcherdidawonderfuljobofmakingthedatacollectionuniform
withuniformcollectionintervals.ClusterHealthMonitorextendsOSwatcherbyensuringitisalwaysscheduledandcollectsdatapointswhileprovidingaclientGUItoviewcurrentload.
Withthisnewtoolweneedtobuyanewdisplaytomonitortheactivitiesoftheclusterinrealtime.
Whydothis?Becauseit’scoolyouhavefullcontrolofyourenvironmentinrealtime.
InthispostI’llshowyouhowtoinstallandconfiguretheIPDClusterMonitor.

2ServersHost(linuxOEL5)alreadywithOracleClusterware/RACinstalled.

1DesktopClient(mylaptop)tomonitortheCluster(usingGUIMode).

Let’sstart.
WhatplatformscanIruntheClusterHealthMonitor?Updated19/07/2011
The
ClusterHealthMonitorisNOTavailableforItaniumplatform(Linux,Windows,andHPItanium)onallversion.

11.2.0.1andearlier:LinuxandWindowsonly(downloadfromOTN)

11.2.0.2:Solaris(Sparcandx86-64)andLinux

11.2.0.3(tobereleased):AIX,Solaris(Sparcandx86-64),Linux,andWindows
TheClusterHealthMonitorisintegratedpartof11.2.0.2OracleGridInfrastructureforLinux(notonLinuxItanium)andSolaris(Sparc64andx86-64only),soinstalling11.2.0.2OracleGridInfrastructureonthoseplatformswillautomatically
installtheClusterHealthMonitor.AIXwillhavetheClusterHealthMonitorstartingfrom11.2.0.3.TheClusterHealthMonitorisalsoenabledforWindows(exceptWindowsItanium)in11.2.0.3.

Installation

ForOTNversionofClusterHealthMonitor,thecompletestepstoinstallthetoolisexplainedinthereadmefileshippedwiththeproduct
For11.2.0.2orlaterversion,theclusterhealthmonitorisinstalledautomaticallywhenGridInfrastructure(akaCRS)isinstalled.TheresourcenameforClusterHealthMonitorisora.crfthatismanagedbyohasd.
WherecanIgetlatestcopyofClusterHealthMonitor?
TheClusterHealthMonitorisintegratedpartof11.2.0.2OracleGridInfrastructureforLinux(notonLinuxItanium)andSolaris(Sparc64andx86-64only),soinstalling11.2.0.2OracleGridInfrastructureonthoseplatformswillautomatically
installtheClusterHealthMonitor.AIXwillhavetheClusterHealthMonitorstartingfrom11.2.0.3.TheClusterHealthMonitorisalsoenabledforWindows(exceptWindowsItanium)in11.2.0.3.
Priorto11.2.0.2onLinuxandpriorto11.2.0.3WindowsexcludingItaniumplatform,theClusterHealthMonitorcanbedownloadedfromOTN.
http://www.oracle.com/technetwork/database/clustering/downloads/index.html
Important:GUI
Mode(AvailableonlywithOTNversion)toversion11.2.0.1.GUIModeisnotavaliableto11.2.0.2
Onlinemodecanbeusedtodetectproblemsliveontheproblemenvironment.ThedatacanbeviewedusingClusterHealthMonitorutility.TheGUIisnotinstalledonthenodesoftheserverbutcanbeinstalledonanyotherclient.
IfyouareusingOracleClusterware11.2.0.2untiltoday(09/08/2011)theGUIModeisnotavaliable.
Ifyouareusing11.2.0.1orpreviousyoumustinstallCHMonServersandClient.
I’llshowyouhowtoperformafullinstallationofCHMonserversandclient.
InstallingCHMonServersLinux
OnLinux,thetoolrequiresLinuxkernelversiongreaterthanorequalto2.6.9andarchitectureisx86.Theinstallwillworkonx86_64aswellifthekernelisconfiguredtorun32-bitbinaries.
1.InLinux,createuser‘:’(e.g.crfuser:oinstall)onallthenodeswheretoolisbeinginstalled.Makesureusername’shomeisthesameonallnodes.Typically,onmostsystems,youwillissue:
Onallnodes:

1	useradd -d /opt/crfuser-s/bin/sh-goinstallcrfuser

2	passwd crfuser

3	Changing password for user crfuser.

4	New UNIXpassword:

5	BAD PASSWORD:itisbasedonadictionaryword

6	Retype newUNIXpassword:

7	passwd : allauthenticationtokensupdatedsuccessfully.

whileloggedinasroot.
2.InLinux,setuppasswordlesssshfortheusercreatedinstep1.Testthatthe”cansshtoallnodes(includingthelocalnode)usinghostname(withoutdomain)withoutpasswordandwithoutanyuserinterventionlikeacknowledgingprompts.
Youcanusethispost:

http://levipereira.wordpress.com/2010/12/07/configure-ssh-for-user-equivalence/

P.SWhenprompt:“Enterpassphrase(emptyfornopassphrase):”type[enter]don’tcreatepassphrase.
TheCHMhaveyourowndatabase.So,youmustspecifythelocationofdatabaseofCHMtostoreOSmetrics.

ThislocationMUSTbeoutsideofthelocationwhereyouunzippedtheZIPfilebecauseallthedirectoriesunderthatlocationwhichwerecreatedbyunzipwillberemoved.

BDBfilescanbekeptasitisforlaterusage.Thelocationshouldbeapathonavolumewithatleast2GBpernodespaceavailableandwritablebyprivilegeduseronly.

ItcannotbeonrootfilesysteminLinux.Thislocationisrequiredtobesameonallhosts.

ThepathMUSTnotbeonshareddisk.IfasharedBDBpathisprovidedtomultiplehosts,BDBcorruptionwillhappen.
Icreateanewdiskof5GBoneachserverandcreateannewVGonLinuxtoStoreDBofCHM.

1	/dev/mapper/VG_ORACRFDB-LV_ORACRFDB

2	4.9G 334M4.1G7%/opt/oracrfdb/db

4.Loginas‘crfuser’onLinux.

Unzipthecrfpack.zipfile.

01	cd /opt/crfuser/ install /

02

$ ls

03	crfpack-linux.zip

04	$ unzipcrfpack-linux.zip

05	Archive: crfpack-linux.zip

06	creating: admin/

07	creating: admin/run/

08

....

09	creating: log/

10	creating: mesg/

5.Runcrfinst.pl(seebelowforusagedetails)scriptonanodewithdesirednodelist,specifiedascommaseparatedlist,forcluster-wideinstall.Youwillfindthisscriptintheinstallsubdirectory.

01	$ cd /opt/crfuser/ install / install /

02	$ ./crfinst.pl-ialemanha,holanda-b/opt/oracrfdb/db-malemanha

03

04	Performing checksonnodes: "alemanha holanda" ...

05	/opt/crfuser/ install doesn't existonholanda,creatingit...

06	Assigning holandaasreplica

07

08	Generating clusterwideconfiguration file ...

09

10	Creating abundle for remote nodes...

11

12	Installing onnodes "holanda alemanha" ...

13

14	Configuration completeonnodes "holanda alemanha" ...

15

16	Please run"/opt/crfuser/ install / install /crfinst.pl -f,optionallyspecifyingBDBlocationwith-basrootoneachnodetocompletethe install process.

17

18

$ su -

6.Oncethestep5finishes,itwillinstructyoutoruncrfinst.plscriptwith-fandoptionally-boneachnodewhileloggedinasroot/admintofinalizetheinstallonthatnode.

01	# /opt/crfuser/install/install/crfinst.pl-f

02	Removing contentsofBDBDirectory/opt/oracrfdb/db

03

04	Installation completedsuccessfullyat/usr/lib/oracrf...

05

06	# sshholanda

07	root@holanda's password:

08	Last login:TueJul1915:41:022011fromalemanha.partnerit.com.br

09	[root@holanda ~] # /opt/crfuser/install/install/crfinst.pl-f

10	Removing contentsofBDBDirectory/opt/oracrfdb/db

11

12	Installation completedsuccessfullyat/usr/lib/oracrf...

7.Enablethetoolonallnodes.Oncethefinalizeoperationiscomplete,run

thefollowingwhileloggedinasprivilegeduser:

1	# /etc/init.d/init.crfdenable,onLinux

toenablethetool.
DONOTbypassanyofabovestepsortryotherwaystoinstallbecausethedaemonswillnotworkcorrectly,andyouwillnotbesupported.
FinishedinstallCHMonServers.
InstallingGUIModeonWindows7x64
StandaloneUIinstallation.OraclerecommendstonotinstalltheUIontheservers.YoucanusethisoptiontoinstalltheUI-onlyclientonaseparatemachineoutsideofcluster.
WemusthaveperlinstalledontheDesktopClienttoinstalltheGUIModeinWindows.
WenotneedtoinstallperlinWindows.WecanusetheperlinstalledonOracleClient.(ifyouinstalledOracleClientonyourdestkop..ofcourse!!!).
AsIhaveinstalledOracleClientonmyDestop,IwillusetheperlofinstallationofOracleClient.
1.DownloadCHMfromOTNforWindows.
2.Unzipthecrfpack.zipfile

3.InstallCHMGUIModeonC:\oracle\product\crf

1	# UsingCMDpromptonWindows

2	c:\> cd C:\Users\Levi\Downloads\crfpack-winnt\ install

3	\> c:\oracle\product\11.2.0\client_1\perl\bin\perl.execrfinst.pl-gc:\oracle\product\crf

4	Installation completedsuccessfullyatc:\\oracle\\product\\crf...

4.SettheenvironmentvariablePERLtoOracleClientonfile“C:\oracle\product\crf\bin\crfgui.bat”

Changethis:

1

...

2	set PERL=perl.exe

3

...

To:

1	set PERL=C:\oracle\product\11.2.0\client_1\perl\bin\perl.exe

NowyoucanuseyourCHMGuiMode,connectonyourcluster.

1	C:\oracle\product\crf\bin>crfgui.bat -m192.168.217.10

2	Cluster HealthAnalyzerV1.10

3	Look for Loggerd vianodealemanha

4	...Connected toLoggerdonalemanha

5	Note: Nodealemanhaisnowup

6	Cluster 'MyCluster' , 2nodes.Ext time =2011-07-19 23:12:58

7	Making Window:IPDClusterMonitorV1.10onmlevi,LoggerV1.04.20091223,Cluster "MyCluster" (View 0),Refreshrate:1sec

Youcanmonitoryourcluster.



InsidetheGUI,youcanuse‘node‘commandtoopenaviewwhichgivesmoredetailedinformationaboutanodeinaNodeView.Alternatively,youcandoubleclickanodetogettheNodeView.

ANodeViewpresentsthedetailedstatisticsoninterestingprocesses,disksandNICsbasedonheuristics.
ClickonHostHolanda…



ClickonHostAlemanha..



OnecandrilldownthedetailsonpartitionsforthediskslistedinNodeViewbydoubleclickingthedisk.TheinformationispresentedintheDiskView.

TheDiskViewprovidesadetailedlistofpartitionsandcorrespondingstatsforeachoneofthem.ItalsoclearlymarkspartitionswhicharefoundtobelongtocertaincategorieslikeVoting/OCR/SWAP/ASMdisks.
BothClusterViewandNodeViewshowtextalertsatthebottom.Thesealertsaregeneratedwhenthesampledvalueofaresourcemetriceithergoesaboveorfallsbelowathresholdthatcouldleadtopotentialproblemsonthenodeand

henceonthecluster.



Youcancheckproblemofnetworkprivate(interconnect)onS.Olevelforexample:
LATENCY(ms)andotherthingsaremonitored.




ThedefaultrefreshrateoftheGUIis1second.Tochangetherefreshrate,use-rwithnumberofseconds(i.e.-r5fora5secondrefreshrate)
C:\oracle\product\crf\bin>crfgui.bat-r5-m192.168.217.10
HistoricalMode
InvokingtheGUIwith‘-d’optionstartsitinhistoricalmode.
C:\oracle\product\crf\bin>crfgui.bat-d“hh:mm:ss”-m192.168.217.10
where-disusedtospecifyhours(),minutes()andseconds()inthepastfromthecurrenttimetostarttheGUIfrome.g.crfgui-d“05:10:00″startstheGUIanddisplaysinformationfromthedatabasewhichis5hoursand

10minutesinthepastfromthecurrenttime.
InvokingtheGUIwith‘-i’optionprovidesthesameshellatthecommandpromptasisseenintheGUIwindowswithapromptof‘toprac>’.Youcanuse‘?’atthisprompttogetdetailedinformationaboutavailablecommandsandoptions.
Enjoy