Dear davey23uk
Thanks for kind help but at the same time I would like to know what are this diag.trc file means and for your kind info I have copied some of its contents ,kindly help me understand the issue

Our's is Rac instance

The output of SQL> show parameter trace is


NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
log_archive_trace integer 0
sql_trace boolean FALSE
trace_enabled boolean TRUE
tracefile_identifier string


This tracve file started generating after we rebounced our all servers including Database

------------------------------------------------------------------------------------------------------------------------

$ more nmh1_diag_9335.trc
/u01/oracle/admin/NMH/bdump/nmh1_diag_9335.trc
Oracle Database 10g Enterprise Edition Release 10.2.0.2.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP and Data Mining options
ORACLE_HOME = /u01/oracle/oracle/product/10.2.0/db
System name: SunOS
Node name: nmhdb1.newmowasat.com
Release: 5.10
Version: Generic_118833-03
Machine: sun4u
Instance name: NMH1
Redo thread mounted by this instance: 0
Oracle process number: 3
Unix process pid: 9335, image: [email protected] (DIAG)

*** SERVICE NAME:() 2009-07-31 05:48:19.123
*** SESSION ID:(334.1) 2009-07-31 05:48:19.123
kjzcprt:rcv port created
Node id: 0
List of nodes: 0,
*** 2009-07-31 05:48:19.125
Reconfiguration starts [incarn=0]
*** 2009-07-31 05:48:19.125
I'm the master node
*** 2009-07-31 05:48:19.125
Reconfiguration completes [incarn=1]
DIAG attached to DLM
cluster reconfiguration is ongoing: 0, 1,
*** 2009-07-31 05:52:11.371
Reconfiguration starts [incarn=2]
*** 2009-07-31 05:52:11.371
I'm the master node
Group reconfiguration cleanup
A rcfg proposal from node 1 is received
*** 2009-07-31 05:52:12.449
Reconfiguration completes [incarn=2]
*** 2009-08-01 11:47:29.200
Switch to short timeout for ipc polling
a session (kjzha) is registered
session (kjzha) is about to end
Registered session (kjzha)[11][4][0][1] is cleaned up
Switch to long timeout for ipc polling
*** 2009-08-01 18:45:19.290
A dump event msg is rcv'd
REQUEST:trace dump in directory cdmp_20090801184035
Trace dumping is performing id=[cdmp_20090801184035]....
*** 2009-08-01 18:46:03.081
Trace dumping is done
*** 2009-08-02 10:11:30.230
Multi-instance trace dumping is performed for a FG process
Performing diagnostic data dump for this instance
Trace dumping is performing id=[cdmp_20090802101130]....
*** 2009-08-02 10:12:15.951
Trace dumping is done
Dump requested by process [orapid=135]
REQUEST:system state dump at level 12
===================================================
SYSTEM STATE
------------
System global information:
processes: base 41744a928, size 300, cleanup 417472d18
allocation: free sessions 41750f538, free calls 0
control alloc errors: 0 (process), 0 (session), 0 (call)
PMON latch cleanup depth: 0
seconds since PMON's last scan for dead processes: 12
system statistics:
14968 logons cumulative
148 logons current
31117505 opened cursors cumulative
2369 opened cursors current
67703 user commits
55062 user rollbacks
163702848 user calls
12234755 recursive calls
668408 recursive cpu usage
$ tail -1000 nmh1_diag_9335.trc
htl=40477c480[40ffe3710,40ff72170] htb=40ff72170 ssga=40ff71518
user=4175dd018 session=4175dd018 count=1 flags=[0000] savepoint=0x3be
LIBRARY OBJECT HANDLE: handle=40afc6078 mtx=40afc61a8(0) cdp=0
name=NMH.ON_APPSVR
hash=6713129f71acce7a8a69a3f9392069b4 timestamp=06-21-2009 17:05:52
namespace=TABL flags=KGHP/TIM/XLR/[00000020]
kkkk-dddd-llll=0000-0011-0015 lock=N pin=0 latch#=3 hpc=0122 hlc=0122
lwt=40afc6120[40afc6120,40afc6120] ltm=40afc6130[40afc6130,40afc6130]
pwt=40afc60e8[40afc60e8,40afc60e8] ptm=40afc60f8[40afc60f8,40afc60f8]
ref=40afc6150[40afc6150,40afc6150] lnd=40afc6168[40afc0258,40af2c818]
LOCK INSTANCE LOCK: id=LB6713129f71acce7a
PIN INSTANCE LOCK: id=NB6713129f71acce7a mode=S release=F flags=[00]
INVALIDATION INSTANCE LOCK: id=IV0001163015120635 mode=S
LIBRARY OBJECT: object=40da30930
type=FNCT flags=EXS/LOC[0005] pflags=NST[0001] status=VALD load=0
DEPENDENCIES: count=3 size=16
READ ONLY DEPENDENCIES: count=1 size=16
ACCESSES: count=1 size=16
DATA BLOCKS:
data# heap pointer status pins change whr
----- -------- -------- --------- ---- ------ ---
0 40afc5fb8 40da30a48 I/-/A/-/- 0 NONE 00
2 40da30608 0 I/-/-/-/- 0 NONE 0c
4 40da30500 40be52ef0 I/-/A/-/- 0 NONE 00
----------------------------------------
SO: 4043bb8d8, type: 53, owner: 4175dd018, flag: INIT/-/-/0x00
LIBRARY OBJECT LOCK: lock=4043bb8d8 handle=416125120 mode=N
call pin=0 session pin=0 hpc=0000 hlc=0000
htl=4043bb958[4043e0638,40ec8dfe0] htb=40ff72320 ssga=40ff71518
user=4175dd018 session=4175dd018 count=1 flags=[0000] savepoint=0x0
LIBRARY OBJECT HANDLE: handle=416125120 mtx=416125250(0) cdp=0
namespace=CRSR flags=RON/KGHP/PN0/EXP/[10010100]
kkkk-dddd-llll=0000-0001-0001 lock=N pin=0 latch#=3 hpc=fffe hlc=fffe
lwt=4161251c8[4161251c8,4161251c8] ltm=4161251d8[4161251d8,4161251d8]
pwt=416125190[416125190,416125190] ptm=4161251a0[4161251a0,4161251a0]
ref=4161251f8[40da67268,40da67268] lnd=416125210[416125210,416125210]
LIBRARY OBJECT: object=40da66e28
type=CRSR flags=EXS[0001] pflags=[0000] status=VALD load=0
DEPENDENCIES: count=3 size=16
AUTHORIZATIONS: count=1 size=16 minimum entrysize=16
ACCESSES: count=1 size=16
TRANSLATIONS: count=1 size=16
SCHEMA: count=3 size=4096 entrysize=0
DATA BLOCKS:
data# heap pointer status pins change whr
----- -------- -------- --------- ---- ------ ---
0 416125060 40da66f40 I/P/A/-/- 0 NONE 00
6 40da67950 40bed8190 I/-/A/-/E 0 NONE 00
----------------------------------------
SO: 4043e05b8, type: 53, owner: 4175dd018, flag: INIT/-/-/0x00
LIBRARY OBJECT LOCK: lock=4043e05b8 handle=416125348 mode=N
call pin=0 session pin=0 hpc=0000 hlc=0000
htl=4043e0638[4045ab788,4043bb958] htb=40ff72320 ssga=40ff71518
user=4175dd018 session=4175dd018 count=1 flags=[0000] savepoint=0x4a825c9b
LIBRARY OBJECT HANDLE: handle=416125348 mtx=416125478(1) cdp=1

------------------------------------------------------------------------------------------------------------------------

Please help me come out of the problem

With Kind regards

Kamal