Recovery War stories???

[roobaron]

Thought this might be interesting...who knows

So what was your worse disaster and how did you recover the database ?

My worst recovery was a period of about 3-4 months where i had to recover at least one database every 2nd day due to a oracle bug.
Hardware: HP
OS: MPE/IX
Ora: 7.2.3
Bug: PMON corrupting datafiles, normally happened late in the day.

The worst thing was we couldn't easily implement archive logging due to dragonian batch job processes which required the logs to be turned off and on and also a lack of free disk space.

So i had the great job of imforming the various clients that unfortunately all the payroll work was lost as we had to restore the previous nites backup. :(

Luckily we finally we able to get most clients on the upgrade path to a sun box and Oracle 8 (Oracle wouldn't fix the problem) and get archive logs happening. We looked like heroes

Have Fun

[musaphura]

Let me tell you one more interesting story

Well there was one guy say Mr X who was a developper. On a conversion process for a major Bank in the banana island, the guy was working till 3:00 in the money. His onversion was stackering because of rollback segments. So, what he did was to commit and truncate the temporary tables as and when required.

There was 4 to 5 seniors behind him. So, he got panicked and drop the transaction table. The whole system went down , can you imagine a whole banking system. So, some of the guys quickly went to the ATMs and draw money since the ATM cannot do a balance checkup for available funds. Wish I did not have a good conscience, otherwise i Would have been rich since i was the one posting the transactions back.

To crown it all, the backups were not up-todate , not because the data was not there, but because we used a pro*C program for calculating month-end balance and the float datatype was not big enough to cater for italian lira.

So, the bottom line is MISFORTUNE NEVER COMES ALONE. The best way is to simulate Crashes on the backup server , waiting for IT to HAPPEN

See you guys