Solaris problems and Unix admin just blew town

The production Oracle box a sunfire 280 is rebooting itself.
Oracle logs show nothing but a sudden restart so that tells me either a power cycle or solaris decided to restart.

Its compounded by another problem which may be totally unrelated.
I got on the console and saw that the year old scripts that predate me and have always worked - are in fact failing. So dbora call to dbstart is bombing as if the wrong shell is involved or the shell executables have become corrupt.

Here is the startup log I captured. I can however manually get Oracle up.

dbora and dbstart exist and have not change since 2002 - any ideas ?

sub-section of console :

> Aug 19 18:22:08 adrastea raid: Sense=700006000000009800000000A0000000000000000
> 000000000000000000000000000000000082C000000000000000000000000000B053154393438323
> 13832392020202020200301040000070000000000000000000000000000000000000000000000000
> 000000000000000000000000000000000000000000000000000000000000000000000000000044F3
> 038313930332F31373536343400000000000000
> Aug 19 18:22:08 adrastea raid: Sense=700006000000009800000000A0000000000000000
> 000000000000000000000000000000000082C000000000000000000000000000B053154393438323
> 13832392020202020200301040000070000000000000000000000000000000000000000000000000
> 000000000000000000000000000000000000000000000000000000000000000000000000000044F3
> 038313930332F31373536343400000000000000
> volume management starting.
> Aug 19 18:22:09 adrastea sendmail[420]: My unqualified host name (adrastea) unkn
> own; sleeping for retry
> Aug 19 18:22:09 adrastea sendmail[482]: My unqualified host name (adrastea) unkn
> own; sleeping for retry
> Aug 19 18:22:09 adrastea sendmail[461]: My unqualified host name (adrastea) unkn
> own; sleeping for retry
> Aug 19 18:22:09 adrastea sendmail[419]: My unqualified host name (adrastea) unkn
> own; sleeping for retry
> Aug 19 18:22:09 adrastea sendmail[417]: My unqualified host name (adrastea) unkn
> own; sleeping for retry
> /etc/rc3.d/S99dbora: [!: not found <--------corrupt shell processor ?
> Sun Microsystems Inc. SunOS 5.9 Generic May 2002
> Aug 19 18:22:12 adrastea sendmail[418]: My unqualified host name (adrastea) unkn
> own; sleeping for retry
> Aug 19 18:22:13 adrastea sendmail[513]: My unqualified host name (adrastea) unkn
> own; sleeping for retry
>


> I believe that is being generated in this Oracle startups script which has not changed so
> perhaps the shell script processor 'sh' has become corrupt ?
>
> if [! -f $ORA_HOME/bin/dbstart] <-- Cannot parse "[!" the file following it exists
> then
> echo "Oracle startup: cannot start"
> exit
> fi


Here is the full console:

> ===================FULL CONSOLE BELOW HERE=======================
>
> wn; sleeping for retry
> my own domain name (galileo) -- using short name
> Aug 19 16:00:01 galileo sendmail[3285]: My unqualified host name (galileo) unkno
> wn; sleeping for retry
> Aug 19 16:01:01 galileo sendmail[3285]: unable to qualify my own domain name (ga
> lileo) -- using short name
> Aug 19 16:01:01 galileo sendmail[3285]: [ID 702911 mail.alert] unable to qualify
> my own domain name (galileo) -- using short name

*****Above repeats and had to be removed

> /dev/rdsk/c2t0d0s3: 7109 files, 60706 used, 10019494 free
> /dev/rdsk/c2t0d0s3: (1158 frags, 1252292 blocks, 0.0% fragmentation)
> The system is coming up. Please wait.
> checking ufs filesystems
> /dev/rdsk/c1t5d5s0: is stable.
> /dev/rdsk/c1t5d7s0: is stable.
> /dev/rdsk/c1t5d6s0: is stable.
> /dev/rdsk/c1t5d3s0: is stable.
> /dev/rdsk/c1t5d4s0: is stable.
> /dev/rdsk/c1t5d2s1: is stable.
> /dev/rdsk/c2t0d0s7: is stable.
> /dev/rdsk/c1t5d0s0: is stable.
> /dev/rdsk/c2t1d0s7: is stable.
> /dev/rdsk/c1t5d1s0: is stable.
> /dev/rdsk/c1t5d2s0: is stable.
> starting rpc services: rpcbind done.
> Setting netmask of eri0 to 255.255.255.0
> Setting default IPv4 interface for multicast: add net 224.0/4: gateway adrastea
> syslog service starting.
> The NVSRAM settings of controller c1t5d0(1T94821829) are correct.
> nvutil command succeeded.
> Array Monitor initiated
> RDAC daemons initiated
> Aug 19 18:22:07 adrastea rdriver: ID[RAIDarray.rdaemon.1001] RDAC Resolution Dae
> mon locked in memory
> Aug 19 18:22:07 adrastea raid: AEN event Host=adrastea Ctrl=1T94821829 Dev=c1t5d
> 0
> Aug 19 18:22:07 adrastea raid: AEN event Host=adrastea Ctrl=1T94821829 Dev=c1t5d
> 0
> Aug 19 18:22:07 adrastea raid: ASC=3F ASCQ=C7 FRU=06 LUN=00 LUN Stat=00
> Aug 19 18:22:07 adrastea raid: ASC=3F ASCQ=C7 FRU=06 LUN=00 LUN Stat=00
> Aug 19 18:22:07 adrastea raid: Sense=7000060000000098000000003FC70600000000000
> 000000000000142000000000000000000080D000000000000000000000000000B053154393438323
> 13832392020202020200301040000000000000000000000000000000000000000000000000000000
> 00000000000000000000000000000000000000000000000000000000000000000000000000004383
> 038313930332F31373536343300000000000000
> Aug 19 18:22:07 adrastea raid: Sense=7000060000000098000000003FC70600000000000
> 000000000000142000000000000000000080D000000000000000000000000000B053154393438323
> 13832392020202020200301040000000000000000000000000000000000000000000000000000000
> 00000000000000000000000000000000000000000000000000000000000000000000000000004383
Sense=700006000000009800000000A0000000000000000
> 000000000000000000000000000000000082C000000000000000000000000000B053154393438323
> 13832392020202020200301040000070000000000000000000000000000000000000000000000000
> 000000000000000000000000000000000000000000000000000000000000000000000000000044F3
> 038313930332F31373536343400000000000000
> Aug 19 18:22:08 adrastea raid: Sense=700006000000009800000000A0000000000000000
> 000000000000000000000000000000000082C000000000000000000000000000B053154393438323
> 13832392020202020200301040000070000000000000000000000000000000000000000000000000
> 000000000000000000000000000000000000000000000000000000000000000000000000000044F3
> 038313930332F31373536343400000000000000
> volume management starting.
> <$view"oracle@adrastea:~>$view /etc/hosts
> "/etc/hosts" [Read only] 16 lines, 573 characters


> *** ns.fsr.com can't find adrastea: Non-existent host/domain
> > exit
> <$view"oracle@adrastea:~>$view /etc/rc3.d/S99dbora
> "/etc/rc3.d/S99dbora" [Read only] 48 lines, 1300 characters
>
> "/etc/rc3.d/S99dbora" [Read only] 48 lines, 1300 characters
> #!/bin/sh
> # Place this file in /etc/init.d
> # Only root can write to /etc/init.d
> # Set ORA_HOME to be equivalent to the $ORACLE_HOME
> # from which you wish to execute dbstart and dbshut;
> #
> # Set ORA_OWNER to the user id of the owner of the
> # Oracle database in ORA_HOME.
> ORA_HOME=/ora/app/oracle/product/9.2
> ORA_OWNER=oracle
> if [! -f $ORA_HOME/bin/dbstart]
> then
> echo "Oracle startup: cannot start"
> exit
> fi
> case "$1" in
> 'start')
> :q
> <$ls"oracle@adrastea:~>$ls
> RMAN_README.TXT dbora rda.tar
> afiedt.buf dumps rman_blprod_bu2.log
> bldemo.html nsmail stdby_installed_prod.txt
> dba_scripts patches
> <$ls"oracle@adrastea:~>$ls -l $ORA_HOME/bin/dbstart
> /bin/dbstart: No such file or directory
> <$ls"oracle@adrastea:~>$ls -l /ora/app/oracle/product/9.2/bin/dbs
> dbshut dbsnmp dbsnmp0 dbsnmpj dbsnmpj0 dbsnmpwd dbstart
> <$ls"oracle@adrastea:~>$ls > -l /ora/app/oracle/product/9.2/bin/dbstart
> -rwxr-xr-x 1 oracle dba 4468 Apr 26 2002 /ora/app/oracle/product/9.
> 2/bin/dbstart
> <$ls"oracle@adrastea:~>$ls -l /bin/sh
> -r-xr-xr-x 4 root root 95488 Apr 6 2002 /bin/sh
> <$"oracle@adrastea:~>$ /etc/rc3.d/S99dbora
> bash: /etc/rc3.d/S99dbora: bad interpreter: Permission denied
> <$"oracle@adrastea:~>$
> úU seeprom format: 0000.0000.0000.0002
> Fatal Error reported by: (PCI-PBM)

Using NIS? Could be the NFS server daemon failed because of problems on that server/machine. Don't know if your sendmail is working or set up, but the message about sleeping for retry and unqualified host name looks like a problem in /etc/defaultdomain, which your file may be okay (all it probably has is ns.fsr.com), but the NIS problem(s) from the error below says something about that type of problem:

> *** ns.fsr.com can't find adrastea: Non-existent host/domain

So if that is the case, then perhaps some of your mount points don't exist (maybe your /etc/init.d directory is automounted on another machine, just a guess, but that would be a strange setup).

Its very strange. The lookup I did failed yet Oracle is up and users are using it - until it decides to spontaneously reboot again.

Tech guy from hardware vendor said :

I had the same 280R that reboot by ifself every one hour or 15 mins. After I install that patch, it was fixed the problem.

Nobody wants to touch the box till admin returns so I am going to do a restore of backups to another box and get users over there. Ironically its the old standby box, which got rebooted, did not auto start, was not detected by monitoring and fell days behind and my RMAN backups have purged the missing logs so I must restore. This was tightly controlled before I was asked to double as the ERP package build an enterprise Microsoft Portal guy. Nonentheless I will have to remember to put my DBA hat on each morning for at least an hour checking all my servers.

Wierdest thing is how the timestamp in the log is wildly all over the place instead of ascending, unless Dataguard was using universal time and server was using local time.