RAC

[Jani]

What are the Real Application Clusters ?
How I could configure these Clusters on Single pc ?
Thanx

[Shestakov]

In first: did u install oracle 9i RAC in separate boxes, if not
try to do it.
I second:
--------------------
How I could configure these Clusters on Single pc ?
--------------------
Probably no ways to do it.
because u must :
-- use rcp, rhost commands in installation process.
-- u should have the same oracle account
(and in this case the same set of enveronment variables)
but in RAC u can't have the same $ORACLE_HOME directory

If u try to use chroot for oracle instances, then we will have the same problem
with oracle account.

If u try to use vertual mashine enveronment, like VMWARE, then u can't
create shared raw devices for db instances, because oracle can't works on
exported or remote mounted file systems.

No way.

[uncleserg]

Yes, you can.
Look at Note 166830.1 Setting up Real Application Cluster (RAC) on Linux - Single node on metalink.oracle.com.
It works...

[Jani]

Dear send me full detail or any url where i found the step by step configuration.

[hrishy]

Hi

Look at Note 166830.1 Setting up Real Application Cluster (RAC) on Linux - Single node on metalink.oracle.com.

i do not have acesses to metalink .could anybody post the contents of the DOC here i would love to try out RAC on my home machine.

regrads
Hrihsy

[sumi_ocp]

there u go..

Doc ID: Note:166830.1
Subject: Setting up Real Application Cluster (RAC) environment on Linux - Single node
Type: BULLETIN
Status: PUBLISHED
Content Type: TEXT/PLAIN
Creation Date: 01-DEC-2001
Last Revision Date: 16-APR-2002




PURPOSE
-------

This document is intended for peoples who would experience Real
Application Cluster (RAC) technology on a common hardware . Basically
this article is based on the same assumptions and limitations discussed
in Note:136077.1 (Setting up Oracle Parallel Server environment on Linux
- Single node [1]), where we argumented the setup for Oracle Parallel
Server (OPS v817).

This article will adopt the key concepts of the above article and
extend them to Oracle RAC 9.0.1 on Linux port .In particular here we will
simplify furthermore the steps necessary to setup a 'RAC-in-a-Box'
environment quickly and easily .


SCOPE & APPLICATION
-------------------
The intended audience, are technicals with a good experience on Linux
operating environment and also introduced to OPS/RAC technology. Here we
will detail the steps necessary to setup the environment and part of
theory behind them .

TITLE
-----

Setting up Real Application Cluster (RAC) on Linux - Single node

CONTENTS
--------

1) Introduction
1.1) DLM, GCS, GES : [2]

2) Architecture
2.1) Two-node / Two-instance RAC Archtecture
2.2) Single-Node / Two-instance RAC Architecture

3) Kernel and System Configuration
3.1) Bill of Materials [3]
3.2) - Prepare your disk for RAW partitions - simplyfied
3.2.1) Cluster Manger RAW Partition can be on FS
3.2.2) GSD Partition (optional)
3.3) Prepare your kernel for RAC single node
3.3.1) KERNEL PARAMETERS relevant to RAC single-node
3.3.2) Hack softdog.c
3.3.3) Kernel memory requirements

4) install Oracle software

5) Setup Oracle Cluster Manager components
5.1) load the softdog module in your kernel at startup
5.2) Create the NM File
5.3) Configure Oracle Cluster Manger (OCM)
5.4) Troubleshooting Oracle Cluster Manger (OCM)
5.4.1) Check filesystem rights
5.4.2) OCM does not start
5.4.3) OCM did not restart
5.4.4) is OCM running ?
5.4.5) how to stop OCM

6) Clustered database setup
6.1) Create RAC database
6.1.1) Database on File System , easyer to setup on FS
6.1.2) create RAC related catalogs
6.1.3) Create extra RLog Threads
6.1.4) Create extra UNDO tablespaces
6.2) setup for use SPFILE
6.3)Configure SPFILE for RAC

7) Startup both the instances in parallel mode
7.1) Start OCM
7.2) setup to unix environment
7.3) startup the instances

8) Switching between RAC and NON-RAC
8.1) Activating RAC feature
8.2) De-activacting RAC feature
8.3) Check if RAC is enabled or not
9) Conclusions

References


1) Introduction
------------

1.1) DLM, GCS, GES : [2]
-------------------

The term Distributed Lock Manager (DLM) is obsolete. The background
processes that were formerly aggregated under the DLM in previous cluster
software products still exist, but the DLM no longer manages resources in
Real Application Clusters. The Global Cache Service (GCS) and Global
Enqueue Service (GES) now handle the management functions that had been
handled by the DLM in earlier Oracle cluster software releases.

Briefly, Global Cache Services (GCS) synchronization layer that allows
lock of resources among all the instances partecipating to the RAC
database in a distributed fashion. This layer is part of the RDBMS kernel
and mantains an inter-instance consistent view of the database of all the
locks and enqueues (GES).

Most of these tasks are carried out by RAC specific background processes,
if compared with [1] Sect 1, we will notice that the bkg. process names
are changed and some of the old process roles has been merged in other :

LMON - (Lock Monitor Process) -> Global Enqueue Service Monitor
The background Global Enqueue Service Monitor (LMON) monitors
the entire cluster to manage global resources. LMON manages
instance and process expirations and the associated recovery
for the Global Cache and Global Enqueue Services. In particular,
LMON handles the part of recovery associated with global
resources. LMON-provided services are also known as cluster
group services (CGS).


LMS - The Global Cache Service Processes (LMSn) handle remote Global
Cache Service messages. Current Real Application Clusters
software provides for up to 10 Global Cache Service Processes.
The number of LMSn processes varies depending on the amount of
messaging traffic among nodes in the cluster. The LMSn processes
handle the acquisition interrupt and blocking interrupt requests
from the remote instances for Global Cache Service resources.
For cross-instance consistent read requests, LMSn creates a
consistent read version of the block and sends it to the
requesting instance. LMSn also controls the flow of messages to
remote instances.

LMD - The Global Enqueue Service Daemon (LMD) is the lock agent process
that manages enqueue manager service requests for Global Cache
Service enqueues to control access to global enqueues and
resources . The LMD process also handles deadlock detection and
remote enqueue requests . Remote enqueue requests are requests
originating from another instance .


LCK - The LCK process has been simplified . Much of the LCK
functionality in previous products is now handled by Global
Cache Service Processes (LMSn). In Real Application Clusters,
there is just one remaining LCK process . It is used for
non-Cache Fusion operations such as instance requests and
cross-instance call operations.

BSP - No longer exist in 9i - Blocks shipping is carried out by GCS
layers provided by LMSn

We point you to [1] Sect.1 for further informations regarding OSD layers
in linux (WD,CM and NM) that are still valid here .

2) Architecture
------------

2.1) Two-node / Two-instance RAC Archtecture
---------------------------------------

Shared Disks,
Database on RAW
partitions
/ \
/ \
Node 1 Node 2
--------------- ---------------
| SGA Instance A| |SGA Instance B |
|---------------| |---------------|
| |pmon|__ | | |pmon|___ |
| |smon |lgwr|| | |smon |lgwr||
| ---|dbwr| | | ---|dbwr| |
| ---- | | ---- |
| ------------| |------------ |
| |lmon lms0 | |lmon lms0 | |
lipc | | lmd | nipc | lmd | | lipc
------- GCS/GES <=================> GCS/GES -------
| | |------------| |------------| | |
| | | CM | | CM | | |
+------> <===================> <------+
| | |___________ | | ___________| | |
| | | NM | | NM | | |
+------> <===================> <------+
| | |___________ | |____________| | |
------> WDT | | WDT <------
| | | | | |
--------------- ---------------
|////kernel/////| |////kernel/////|
--------------- ---------------

'-' lipc : local inter-process communication
'=' nipc : networked inter-process communication (via tcp/ip)

Note : Typically the shared disks box is wired via SCSI or Optical
Fiber to the main boards, this introduce compatibility issues related
to the drivers used and to the controllers used . Please refer to HW
certification matrices for that.

2.2) Single-Node / Two-instance RAC Architecture
-------------------------------------------

Local Disk,
Database on FILE SYSTEM

|
single Node |
---------------------------------------------
| SGA Instance A| |SGA Instance B |
|--------------- ---------------|
| |pmon|__ |pmon|___ |
| |smon |lgwr| |smon |lgwr||
| ---|dbwr| ---|dbwr| |
| ---- (localhost) ---- |
| __________ nipc __________ |
| | GCS/GES | <=========> | GCS/GES | |
| | lmon | | lmon | |
| | lmd lms0 | | lmd lms0 | |
| |__________| |__________| |
| ____|________________________|_____ |
| | | | | |
| | +--> CM svcs <--+ | |
| |____|________________________|_____| |
| | | | | |
| | +--> NM svcs <--+ | |
| |____|________________________|_____| |
| | | | | |
| | --> WDT svcs <-- lipc | |
---------------------------------------------
|//////////////// kernel ////////////////////|
---------------------------------------------

'-' lipc : local inter-process communication
'=' nipc : networked inter-process communication (via tcp/ip)

Note : this kind of configuration is not so exposed to
compatibility issues related to various SCSI controller and
drivers that may limit the implementations .

Note : we will use normal ext2 file system, see below for
further details (3.2) .



3) Kernel and System Configuration
--------------------------------
we tested on Dell latitude pIII 500Mhz - 12Gb IDE - 256MB ram - CDROM)


3.1) Bill of Materials [3]
-------------------

Intel x86 X-windowed and networked box
SuSE 7.2,Red Hat 7.1 distributions
Related Linux kernel source 2.4 rpms (kernel-2.4.XX-YY.i386.rpm)
Oracle 9.0.1.0. for Linux Intel 32bit

Operating System Linux : kernel 2.4
Operating System Patches: Patch 4 or later
Raw should be included in above distr.
System Libraries : GNU Lib C 2.2,make-3.79,binutils-2.10

Disk Space Requirements : ~2GB for an average install, Over 2.5GB for
Default installation .


3.2) - Prepare your disk for RAW partitions - (simplyfied)
--------------------------------------------------------
In [1] we showed that every database file was positioned on a dedicated
RawDevice (RD) , and this common in a multinode RAC environments to
override dangerous unsyncronized File System caches . In a single-node
configuration we can remove the RD constraint because FS caches
coerency issues does not arise us when the FS caches involved is
just one .


3.2.1) Cluster Manger RAW Partition can be on FS
-----------------------------------------
Node Monitor needs a raw partition to use as syncronization/voting aera
among nodes partecipating the cluster . This partition can be a file in
force of the single node hipotesys and the size should be at least :

4 + [(the number of nodes in the cluster) * 4] KB [1]

We will use 1M for our scopes (see 5.2)


3.2.2) GSD Partition (optional)
------------------------
If you plan to use 'SRVCTL' utility [4] then you will need a RawDevice
partition of about 100M to let GSD (Global Services Daemon) work [5].
Infact this daemon checks (with an iostat primitive) that the file
descriptor points really to a RD .

infact

in : $ORACLE_HOME/srvm/config
there is : 'srvConfig.loc'
that contains : '/u01/ora/oradata/RAC_Cluster_Configuration_File'
that should point to your RD .

if you issue for example :

% gsd
% srvconfig -version

will raise ->

"oracle.ops.mgmt.rawdevice.RawDeviceException: PRKR-1020 :
reading of version information failed,
/u01/ora/oradata/RAC_Cluster_Configuration_File
is not a raw device file, No such file or directory"

To avoid this you must provide a true Raw Partition .



3.3) Prepare your kernel for RAC single node
----------------------------------------
In [1] we discussed in this section how to build the kernel and which
parameters define in order to allow OPS work . Here we will remove some
constraints in order to let your RAC environment be more usable for the
scopes of this document .

- we assume that you have kernel sources in place and you
are familiar with linux kernel setup (see [1] Sect-3.3 for hints)

3.3.1) KERNEL PARAMETERS relevant to RAC single-node
---------------------------------------------

'Character Devices' -> 'Watchdog Cards' :

CONFIG_WATCHDOG
'Watchdog Timer Support' = 'y'

CONFIG_SOFT_WATCHDOG
'Software Watchdog' = 'm'

CONFIG_WATCHDOG_NOWAYOUT
'Disable watchdog shutdown on close' = 'n' (*)

-----------------
*Important Note* :
-----------------
RAC multinode needs 'y' [1] , infacts, the default watchdog
behaviour (which you get if you say N here) is to stop the
timer if the process managing it closes the file /dev/watchdog.
It's always remotely possible that this process might get
killed. If you say Y here, the watchdog cannot be stopped once
it has been started . In our single-node environment is not
unfrequent that 'watchdogd' hangs when OCM crash. In these
cases we have to stop and restart, for this reason we use 'n'
in order to avoid us to be forced to bounce the entire box when
'watchdogd' stucks .


3.3.2) Hack softdog.c
--------------
The software watchdog 'softdog' that we use in our environment have a
pourpose in life : reboot the node when one of the registered processes
forget to 'ping' watchdogd . This is reasonable multi-node RAC, because
prevents that the entire cluster hangs due to fact that a single remote
key processes (LMON..) does not respond to posts because is dead or is
spinning . It is preferrable with linux port, to loose the node and
reconfigure the cluster and let the system proceede .

This happens for example, in case of an instance failure. The processes
belonging to that RAC instance typically will not de-register correctly
in case of a crash and softdog will trigger the node reboot.In our case,
this is not so good, because we have more than one instance on the same
node and we expect that we can continue with the surviving ones. For
this reason we will modify the source code of the softdog, 'softdog.c',
in order to skip the reboot (*)

Currently, we observed that in certain types of sudden crashes we loose
also OCM and conseguently the other instances goes down , this is
normal and is tolerated within the limits of this special environment .


(become root, we use kernel label '2.4.7', adapt to your one)

% cd /usr/src/linux-2.4.7/drivers/char
% cp softdog.c softdog.c.orig #-- make a backup copy


modify softdog.c in this way, adding the line at the
beginning of the file :

#define ONLY_TESTING

% cd /usr/src/linux-2.4.7
% make modules


wait until all modules are built (typically unless you performed
a 'clean' all obj should be already in place and only our softdog
will be rebuilt . Check that the module was rebuilt :

% ls -l /usr/src/linux-2.4.7/drivers/char/softdog.o

check if softdog is already loaded, shutdown all his clients and
unload it :

% lsmod

Module Size Used by
...
softdog 1504 1 <--- in this case YES
autofs 10944 1 (autoclean)
serial_cb 1632 1
cb_enabler 2688 2 [serial_cb]
...

% rmmod softdog

substite the newly one :

% cd /lib/modules/2.4.7/kernel/drivers/char
% cp softdog.o softdog.o.orig #-- back it up
% cp /usr/src/linux-2.4.7/drivers/char/softdog.o .

now you can 'insmod' it again


-----------------
*Important Note* :
-----------------
RAC on Linux does not allow the system to continue upon a key
process failure and the softdog.c hack has to be intended
finalyzed to scopes and limitations of this article .

>>Modifying the sofdog code when using RAC is not supported <<

3.3.3) Kernel memory requirements
--------------------------
Oracle 9i naturally is more memory consumer than previously
releases and you have to instruct linux kernel to allow bigger
memory mappings.

The followings are a suitable configuration, merge it in your
/etc/rc.d/rc.local :


#--- Oracle 9i needed : rc.local PART 1/2
cd /proc/sys/kernel

cat sem
export SEMMSL=200
export SEMMNS=512
export SEMOPM=200
export SEMMNI=200

echo $SEMMSL $SEMMNS $SEMOPM $SEMMNI > sem

cat shmmax shmmni shmmni shmseg
export SHMMAX=2147483648
export SHMMIN=1
export SHMMNI=100
export SHMSEG=4096

echo $SHMMAX > shmmax
echo $SHMMIN > shmmin
echo $SHMMNI > shmmni
echo $SHMSEG > shmseg

cat shmmax shmmni shmmni shmseg
#--- End of rc.local PART 1/2



4) install Oracle software
-----------------------
Simply install the software as usual taking in care of explicitly select :

'Oracle Real Application Clusters 9.0.1.0.0'

from the software list in addition to the other options you will need .

This will install :

Oracle Cluster Manager 2.0.0.10.0 (in $ORACLE_HOME/oracm)
Oracle RAC Configuration Tool

5) Setup Oracle Cluster Manager components
---------------------------------------

5.1) load the softdog module in your kernel at startup
-------------------------------------------------
appending to your system startup file the :

#--- Oracle 9i needed : rc.local PART 2/2
# (this command must be done by root)

/sbin/insmod softdog soft_margin=60

#--- End of rc.local PART 2/2


5.2) Create the NM File
------------------
we will use a 1M file that is enough

% dd if=/dev/zero of=/u01/ora/oradata/RAC_Node_Monitor_file \
bs=1024 count=1024

1024+0 records in
1024+0 records out

5.3) Configure Oracle Cluster Manger (OCM)
-------------------------------------
like for OPS, in RAC for linux the OSD layer is developped by Oracle and
is located under $ORACLE_HOME/oracm . Here you will find :

$ORACLE_HOME/oracm/
admin/ -> CM configuration files resides :
nmcfg.ora -> Node Monitor conf. file
ocmargs.ora -> OCM startup parameters

bin/ -> the modules :
ocmstart.sh -> OCM startup script
oracm -> CM binary
oranm -> NM binary
watchdogd -> WD binary

rawio rdevtest -> troubleshooting tools
wdttest1 wdttest2

log/ -> where the logs will go :
cm.log cm.out -> CM log stream and stdout
nm.log nm.out -> NM log stream and stdout
ocmstart.ts -> ocmstart.sh lock file
wdd.log -> WD out

configure the NM :

edit 'nmcfg.ora' and substitute the 'DefinedNodes' with
your `hostname` , and 'CmDiskFile' with the file defined
above (5.2)

DefinedNodes=cclinux
CmDiskFile=/u01/ora/oradata/RAC_Node_Monitor_file

Note : the node name should be returned by 'hostname' and
resolved in /etc/hosts (if you're using DHCP you may set the
/etc/hosts alias for 127.0.0.1 to the output of `hostname`).
This will prevent unnecessary IP roundtrips .


configure OCM :

edit 'ocmargs.ora' and check the following defaults

watchdogd -g dba
oranm
oracm
norestart 1800

if you take a look to 'ocmstart.sh' you will understand the
meaning of each OCM parameters


5.4) Troubleshooting Oracle Cluster Manger (OCM)
-------------------------------------------
may happen that OCM won't start, the reasons should be investigated
analyzing the log files of NM,CM,WD .

5.4.1) Check filesystem rights
-----------------------
When starting OCM , 'ocmstart.sh' may return something like :

oranm: cnmrequest.c:138: ConnectListener: Assertion `0' failed

the 'nm.log' shows :
...
WARNING | ConnectListener(): failed to remove /tmp/.oranmsock errno =1
MESSAGE | failed to bind at /tmp/.oranmsock errno=98
...

infact in this case :

% ls -l /tmp/.ora*sock*
srwxrwxrwx 1 root bin 0 Dec 3 11:23 /tmp/.oracmsock
srwxr-xr-x 1 root bin 0 Dec 3 11:23 /tmp/.oranmsock

(stop all OCM and remove them , they will be re-created)

If you start OCM with 'oracle' account, then verify that you
have some FS rights to do that .

/tmp/.watchdog/* - directory where resides the sockets used
by 'oracm','oranm', 'lmon' <-> 'watchdogd'
in the form cl_sock__XXXX
/tmp/.oranmsock - socket for NM
/tmp/.oracmsock - socket for CM

/u01/ora/product/9.0.1/oracm/log/ocmstart.ts - Lock semaphore

5.4.2) Oracle does not start
---------------------
typically when facing errors like this during the startup

SQL> startup
CM:CmThreadAttach: socket not connected (error=111)

you should check that all modules of OCM are running, if not, or
only a part of OCM is running , stop all OCM modules and restart
them again


5.4.3) OCM did not restart
-------------------
if you experience errors like :

ocmstart.sh: Error: Restart is too frequent
Info: Check the system configuration and fix the problem.
Info: After you fixed the problem, remove the timestamp file
Info: "/u01/ora/product/9.0.1/oracm/log/ocmstart.ts"

and OCM is not running , you need to remove the semaphore file
indicated .

% rm /u01/ora/product/9.0.1/oracm/log/ocmstart.ts

5.4.4) is OCM running ?
-----------------
a quick way to check if OCM is running is to check if you can see a
similar processes tree:

watchdogd -g dba
oranm
\_ oranm
\_ oranm
\_ oranm
\_ oranm
\_ oranm
\_ oranm
oracm
\_ oracm
\_ oracm

5.4.5) how to stop OCM
---------------
basically you have to kill all the above OCM related processes, after
you determe that there are not any RAC instance is running .

%
%
%


6) Clustered database setup
-----------------------
At this point our system is ready to start a couple of instances on the
same database, but before that, we have to create the database. A database
that is intended to be opened in cluster mode is quite the same that one
in exlusive mode, except for some main issues .

- SPFILE.ora : In a clustered database we will have an specific section
for each instance that will mount the database . With 9i, we introduced
the SPFILE (Server Parameter FILE) that store all the instance parameters.
SPFILE's command syntax is CLUSTER-Awared and we can specify all our needs
in a single file . With 9i, in a multi-nodes RAC configuration, SPFILE can
be efficiently pointed to a shared RD partition simplyfiing the setup and
avoiding parameter misconfiguration .

- Redo Logs : In a clustered database each instance must have his own Redo
Logs, so we will talk of 'Redo Log Threads' (RLT). RLT are numbered from 1
to n and normally we have only one thread of Redo Logs and this is the RLT
#1. In RAC each starting instance will use exclusively the RLT number 'n'
where 'n' is equal to the 'thread=n' parameter specific for that istance .

- Rollback Segments : Each instance must acquire at least one private RBS,
wich one is controlled with instance specific 'rollback_segment=(....)'
parameter. This is the pre-9i way to define the undo space for OPS , 9i
introduced the concept of Self Managed UNDO Space, through the concept of
UNDO tablespace built in the database. In a RAC environment, you should
setup an UNDO tablespace for every instance you plan to join the database.


6.1) Create RAC database
-------------------
DataBase Configuration Assistan (DBCA) can be used to build the scripts
that define our database . DBCA is a RAC-Awared utility, in therms that it
can drive you through the creation of a clustered database but to do that
require it an extra setup not covered by this document (see GSD,srvconfig,
srvctl .. [4],[5]) .

We prefer to create a normal database as we are used, then 'transform' it
in a 'clusterizable' one . In this way we think that we will understand
better some concepts that can be helpfull if using more sophisticated
tools like DBCA in the future .


6.1.1) Database on File System , easyer to setup on FS
---------------------------------------------
In force of above hipotesys (3.2) we will use normal File System
datafiles (*) simplyfing the environment we are going to setup

DataBase 'NINEI'
----------------
Create it using your preferred script/tool ...

PATH |Description
----------------------------------------|---------------
$ORACLE_BASE/oradata/NINEI/
control01.ctl control02.ctl <- Controlfiles
+) redo11.log +) redo12.log <- Thread#1 RLogs
+) redo21.log +) redo22.log <- Thread#2 RLogs
system01.dbf
temp01.dbf
tools01.dbf
users01.dbf
undotbs01.dbf <- UNDO TS Instance 1 (20M)
+) ora_undotbs__y0hrj0og.dbf <- UNDO TS Instance 2 (20M)
(autogenerated)

$ORACLE_BASE/oradata/
RAC_Node_Monitor_file <- Node Monitor


Note : the '+' marked files are created by the next steps .

Note : as you can see is quite like a normal test database and
the respective sizes are also nothing special . In following
sections we will assume that you start from a standard database
created with DBCA using OFA statements.


-----------------
*Important Note* :
-----------------
DO NOT USE File System based RAC databases in a multi-node
environment unless it is explicity certified and supported (i.e.
Tru64 RAC or OpenVMS RAC ..)



6.1.2) create RAC related catalogs
---------------------------

% cd $ORACLE_HOME/rdbms/admin
% sqlplus "/ as sysdba"

@catparr.sql


6.1.3) Create extra RLog Threads
-------------------------
once the database is started exclusive the first time, we can
create the RLT for both the instances :

%sqlplus "/ as sysdba"

ALTER DATABASE ADD LOGFILE THREAD 1 GROUP 11
'/u01/ora/oradata/NINEI/redo11.log size' 10M ;
ALTER DATABASE ADD LOGFILE THREAD 1 GROUP 12
'/u01/ora/oradata/NINEI/redo12.log size' 10M ;
ALTER DATABASE ADD LOGFILE THREAD 2 GROUP 21
'/u01/ora/oradata/NINEI/redo21.log size' 10M ;
ALTER DATABASE ADD LOGFILE THREAD 2 GROUP 22
'/u01/ora/oradata/NINEI/redo22.log size' 10M ;
ALTER DATABASE ENABLE THREAD 2;

(optionally use the 'reuse' clause)

6.1.4) Create extra UNDO tablespaces
-----------------------------
every 9i RAC instance needs his own UNDO tablespace when using
undo_management=AUTO :

%sqlplus "/ as sysdba"

create UNDO tablespace UNDOTBS_B datafile size 20M;

6.2) setup for use SPFILE
--------------------
Assuming that you created a database named 'NINEI' and an instance
named 'NINEI' for normal non-clustered operations, we are going to
create the SPFILE that will used by all the instances when mounting
clustered database .

normally is (or check that it is) :

% cd $ORACLE_HOME/dbs
% ls -l initNINEI.ora

initNINEI.ora -> /u01/ora/admin/NINEI/pfile/initNINEI.ora

where 'initNINEI.ora' contains all parameters, now we create a new SPFILE
to use with RAC starting from this file

% sqlplus "/ as sysdba"

create spfile='/u01/ora/admin/NINEI/pfile/spfileNINEI.ora.RAC'
from pfile='u01/ora/admin/NINEI/pfile/initNINEI.ora';

now save a backup copy of you initNINEI.ora and edit . modify it
in order to contain only the following line :

spfile=/u01/ora/admin/NINEI/pfile/spfileNINEI.ora.RAC

pointing to our new SPFILE . Verify that shutting down the instance and
then starting up all is working with SPFILE .


6.3)Configure SPFILE for RAC
------------------------
the following steps assume that we will start the clustered database from
two instances named 'RAC_A' and 'RAC_B', errors like :

'ORA-32010: cannot find entry to delete in SPFILE '

are expected and can be ignored . Note also that the order to run
these statements is relevant ..

Key
Point | RAC Parameter
------|-----------------------------------------------------
(1)
alter system reset cluster_database scope=spfile sid='*';
alter system set cluster_database=TRUE scope=spfile sid='RAC_A';
alter system set cluster_database=TRUE scope=spfile sid='RAC_B';

(2)
alter system reset instance_name scope=spfile sid='*';
alter system set instance_name=RAC_A scope=spfile sid='RAC_A';
alter system set instance_name=RAC_B scope=spfile sid='RAC_B';

(3)
alter system reset undo_tablespace scope=spfile sid='*';
alter system set undo_tablespace=UNDOTBS scope=spfile sid='RAC_A';
alter system set undo_tablespace=UNDOTBS_B scope=spfile sid='RAC_B'

(4)
alter system reset instance_number scope=spfile sid='*';
alter system set instance_number=1 scope=spfile sid='RAC_A';
alter system set instance_number=2 scope=spfile sid='RAC_B';

(5)
alter system reset thread scope=spfile sid='*';
alter system set thread=1 scope=spfile sid='RAC_A';
alter system set thread=2 scope=spfile sid='RAC_B';

(6)
alter system reset cluster_database_instances scope=spfile sid='*';
alter system set cluster_database_instances=2 scope=spfile sid='*';

(7) [optional if not in ArchiveLogMode]
alter system reset log_archive_dest_1 scope=spfile sid='*';
alter system reset log_archive_dest scope=spfile sid='*';
alter system set
log_archive_dest_1='location=/u01/ora/oradata/NINEI/logA'
scope=spfile sid='RAC_A';
alter system set
log_archive_dest_1='location=/u01/ora/oradata/NINEI/logB'
scope=spfile sid='RAC_B';
alter system reset log_archive_format scope=spfile sid='*';
alter system set
log_archive_format='log_%t_%s' scope=spfile sid='*';

(8) [optional if not using TAF features]

alter system reset service_names scope=spfile sid='*';
alter system set service_names='NINEI,SERVICE_A'
scope=spfile sid='RAC_A';
alter system set service_names='NINEI,SERVICE_B'
scope=spfile sid='RAC_B';


Key Point Considerations :
------------------------
(1) We explicity set 'CLUSTER_DATABASE=TRUE' for each instance
and not with the 'sid=*' qualifier in order to allow us to start
the 'NINEI' database in exlusive mode if needed . It will be
enough to switch to the orginal 'NINEI' instance environment
used for the creation .

(2) Usefull for service registration and OEM .

(3) Every instance have his own UNDO tablespace

(4),(5) Obvious

(6) In 9i RAC, this parameter is used to dimensionate the inner
kernel structures that lay under GCS/GES, it's used for DLM Fast
Reconfiguration algorithms (Lazy Remastering). It should be
setted to the number of potential instances joining the
clustered database .

(7) RAC instances normally keep different archive log destinations

(8) used for TAF configuration & tests




7) Startup both the instances in shared mode
-----------------------------------------
Assuming that your oracle is linked with RAC enabled, we perform the
following task from the 'oracle' unix account .

7.1) Start OCM
---------
login with you 'oracle' account


check that softdog is in place (5.1)

% /usr/sbin/lsmod | grep softdog

start OCM with the new provided script

% cd $ORACLE_HOME/oracm/bin
% ./ocmstart.sh



7.2) setup unix environment
-------------------------
in order to have the ORACLE_HOME pointing to the 9.0.1.0.0 code path
and two distict ORACLE_SID :

Instance RAC_A Instance RAC_B
-------------- --------------

ORACLE_SID=RAC_A ORACLE_SID=RAC_B
ORACLE_BASE=/u01/ora ORACLE_BASE=/u01/ora
ORACLE_HOME=/u01/ora/product/9.0.1 ORACLE_HOME=/u01/ora/product/9.0.1
ORACLE_TERM=vt100 ORACLE_TERM=vt100
PATH=$PATH:$ORACLE_HOME/bin: PATH=$PATH:$ORACLE_HOME/bin:
PATH=$PATH:$ORACLE_HOME/oracm/bin PATH=$PATH:$ORACLE_HOME/oracm/bin
LD_LIBRARY_PATH=$ORACLE_HOME/lib LD_LIBRARY_PATH=$ORACLE_HOME/lib
export ORACLE_BASE export ORACLE_BASE
export ORACLE_HOME export ORACLE_HOME
export ORACLE_TERM export ORACLE_TERM
export ORACLE_SID export ORACLE_SID
export PATH export PATH
export LD_LIBRARY_PATH export LD_LIBRARY_PATH

7.3) startup the instances
---------------------
The usual symbolic links to init files should already be created in
$ORACLE_HOME/dbs .

initRAC_A.ora -> /u01/ora/admin/NINEI/pfile/initNINEI.ora
initRAC_B.ora -> /u01/ora/admin/NINEI/pfile/initNINEI.ora

Note : we point to the same init.ora that contains the pointing
to the same shared SPFILE .


% sqlplus "/ as sysdba" % sqlplus "/ as sysdba"
startup startup
... ...

if all worked ok, both the istances will startup, monitor
'alertRAC_A.log' , 'alertRAC_B.log' for errors

the process view should show the follwing process runnings :


ora_pmon_RAC_A ora_pmon_RAC_B
ora_diag_RAC_A ora_diag_RAC_B
ora_lmon_RAC_A ora_lmon_RAC_B
ora_lmd0_RAC_A ora_lmd0_RAC_B
ora_lms0_RAC_A ora_lms0_RAC_B
ora_lms1_RAC_A ora_lms1_RAC_B
ora_dbw0_RAC_A ora_dbw0_RAC_B
ora_lgwr_RAC_A ora_lgwr_RAC_B
ora_ckpt_RAC_A ora_ckpt_RAC_B
ora_smon_RAC_A ora_smon_RAC_B
ora_reco_RAC_A ora_reco_RAC_B
ora_arc0_RAC_A ora_arc0_RAC_B
ora_lck0_RAC_A ora_lck0_RAC_B


8) Switching between RAC and NON-RAC
---------------------------------
You can switch back your oracle to a NON-RAC mode relinking the executable.
If you use the default 'NINEI' SID name you can point to the same SPFILE .

8.1) Activating RAC feature
----------------------
If you want to open your database in SHARED MODE (CLUSTER_DATABASE=TRUE)
you MUST have oracle RAC-ENABLED

% cd $ORACLE_HOME/rdbms/lib
% make -f ins_rdbms.mk rac_on
% make -f ins_rdbms.mk ioracle

you can however open a database with CLUSTER_DATABASE=FALSE using a RAC
ENABLED oracle .

8.2) De-activacting RAC feature
--------------------------
If you want to open any database without OCM running you MUST have oracle
RAC-DISABLED .

% cd $ORACLE_HOME/rdbms/lib
% make -f ins_rdbms.mk rac_off
% make -f ins_rdbms.mk ioracle

8.3) Check if RAC is enabled or not
------------------------------
In linux (only) you can use the follwing statement in order to chech if
an Oracle RAC specific library is linked against the current executable

% ldd $ORACLE_HOME/bin/oracle | grep libwddapi.so

IF <1 row> IS RETURNED
THEN your oracle IS RAC-ENABLED
ELSE your oracle IS NOT RAC-ENABLED




9) Conclusions
-----------
The solution exposed here looses most of the key features of a RAC
solution that depends on multiple nodes to scale-up application
performances and to be Highly Available eliminating single points of
failures such instances and nodes .

This document should be intended principally for didactic scopes and used
by peoples that are testing the RAC technology and need to setup a
quick solution for simulation, test and developement purposes .


The 'RAC-in-a-box' solution can be suggested also for :

- estimate scaling factor curve for an application with RAC
- scale up on a single node with multiple SGAs in case of
very high latch contention issues
- more than 2 clustered instances, we tested with up to 6 RAC
instances on a laptop



-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --

RELATED DOCUMENTS
-----------------


[1] Note:136077.1 Setting up Oracle Parallel Server
environment on Linux - Single node

[2] Oracle9i RAC - Concepts (9.0.1)

[3] Oracle9i Release Notes (9.0.1) for Linux Intel

[4] Note:159371.1 SRVCTL utility in Oracle9I RAC

[5] Oracle9i RAC - Installation and Configuration