RAC
DBAsupport.com Forums - Powered by vBulletin
Results 1 to 6 of 6

Thread: RAC

  1. #1
    Join Date
    Aug 2001
    Posts
    134

    Thumbs down

    What are the Real Application Clusters ?
    How I could configure these Clusters on Single pc ?
    Thanx


  2. #2
    Join Date
    Sep 2001
    Location
    NJ, USA
    Posts
    1,287
    In first: did u install oracle 9i RAC in separate boxes, if not
    try to do it.
    I second:
    --------------------
    How I could configure these Clusters on Single pc ?
    --------------------
    Probably no ways to do it.
    because u must :
    -- use rcp, rhost commands in installation process.
    -- u should have the same oracle account
    (and in this case the same set of enveronment variables)
    but in RAC u can't have the same $ORACLE_HOME directory

    If u try to use chroot for oracle instances, then we will have the same problem
    with oracle account.

    If u try to use vertual mashine enveronment, like VMWARE, then u can't
    create shared raw devices for db instances, because oracle can't works on
    exported or remote mounted file systems.

    No way.


  3. #3
    Join Date
    Apr 2002
    Posts
    9
    Yes, you can.
    Look at Note 166830.1 Setting up Real Application Cluster (RAC) on Linux - Single node on metalink.oracle.com.
    It works...

  4. #4
    Join Date
    Aug 2001
    Posts
    134

    uncleserg

    Dear send me full detail or any url where i found the step by step configuration.

  5. #5
    Join Date
    Jan 2001
    Posts
    2,828

    Talking

    Hi

    Look at Note 166830.1 Setting up Real Application Cluster (RAC) on Linux - Single node on metalink.oracle.com.

    i do not have acesses to metalink .could anybody post the contents of the DOC here i would love to try out RAC on my home machine.

    regrads
    Hrihsy

  6. #6
    Join Date
    Mar 2002
    Posts
    60
    there u go..

    Doc ID: Note:166830.1
    Subject: Setting up Real Application Cluster (RAC) environment on Linux - Single node
    Type: BULLETIN
    Status: PUBLISHED
    Content Type: TEXT/PLAIN
    Creation Date: 01-DEC-2001
    Last Revision Date: 16-APR-2002




    PURPOSE
    -------

    This document is intended for peoples who would experience Real
    Application Cluster (RAC) technology on a common hardware . Basically
    this article is based on the same assumptions and limitations discussed
    in Note:136077.1 (Setting up Oracle Parallel Server environment on Linux
    - Single node [1]), where we argumented the setup for Oracle Parallel
    Server (OPS v817).

    This article will adopt the key concepts of the above article and
    extend them to Oracle RAC 9.0.1 on Linux port .In particular here we will
    simplify furthermore the steps necessary to setup a 'RAC-in-a-Box'
    environment quickly and easily .


    SCOPE & APPLICATION
    -------------------
    The intended audience, are technicals with a good experience on Linux
    operating environment and also introduced to OPS/RAC technology. Here we
    will detail the steps necessary to setup the environment and part of
    theory behind them .

    TITLE
    -----

    Setting up Real Application Cluster (RAC) on Linux - Single node

    CONTENTS
    --------

    1) Introduction
    1.1) DLM, GCS, GES : [2]

    2) Architecture
    2.1) Two-node / Two-instance RAC Archtecture
    2.2) Single-Node / Two-instance RAC Architecture

    3) Kernel and System Configuration
    3.1) Bill of Materials [3]
    3.2) - Prepare your disk for RAW partitions - simplyfied
    3.2.1) Cluster Manger RAW Partition can be on FS
    3.2.2) GSD Partition (optional)
    3.3) Prepare your kernel for RAC single node
    3.3.1) KERNEL PARAMETERS relevant to RAC single-node
    3.3.2) Hack softdog.c
    3.3.3) Kernel memory requirements

    4) install Oracle software

    5) Setup Oracle Cluster Manager components
    5.1) load the softdog module in your kernel at startup
    5.2) Create the NM File
    5.3) Configure Oracle Cluster Manger (OCM)
    5.4) Troubleshooting Oracle Cluster Manger (OCM)
    5.4.1) Check filesystem rights
    5.4.2) OCM does not start
    5.4.3) OCM did not restart
    5.4.4) is OCM running ?
    5.4.5) how to stop OCM

    6) Clustered database setup
    6.1) Create RAC database
    6.1.1) Database on File System , easyer to setup on FS
    6.1.2) create RAC related catalogs
    6.1.3) Create extra RLog Threads
    6.1.4) Create extra UNDO tablespaces
    6.2) setup for use SPFILE
    6.3)Configure SPFILE for RAC

    7) Startup both the instances in parallel mode
    7.1) Start OCM
    7.2) setup to unix environment
    7.3) startup the instances

    8) Switching between RAC and NON-RAC
    8.1) Activating RAC feature
    8.2) De-activacting RAC feature
    8.3) Check if RAC is enabled or not
    9) Conclusions

    References


    1) Introduction
    ------------

    1.1) DLM, GCS, GES : [2]
    -------------------

    The term Distributed Lock Manager (DLM) is obsolete. The background
    processes that were formerly aggregated under the DLM in previous cluster
    software products still exist, but the DLM no longer manages resources in
    Real Application Clusters. The Global Cache Service (GCS) and Global
    Enqueue Service (GES) now handle the management functions that had been
    handled by the DLM in earlier Oracle cluster software releases.

    Briefly, Global Cache Services (GCS) synchronization layer that allows
    lock of resources among all the instances partecipating to the RAC
    database in a distributed fashion. This layer is part of the RDBMS kernel
    and mantains an inter-instance consistent view of the database of all the
    locks and enqueues (GES).

    Most of these tasks are carried out by RAC specific background processes,
    if compared with [1] Sect 1, we will notice that the bkg. process names
    are changed and some of the old process roles has been merged in other :

    LMON - (Lock Monitor Process) -> Global Enqueue Service Monitor
    The background Global Enqueue Service Monitor (LMON) monitors
    the entire cluster to manage global resources. LMON manages
    instance and process expirations and the associated recovery
    for the Global Cache and Global Enqueue Services. In particular,
    LMON handles the part of recovery associated with global
    resources. LMON-provided services are also known as cluster
    group services (CGS).


    LMS - The Global Cache Service Processes (LMSn) handle remote Global
    Cache Service messages. Current Real Application Clusters
    software provides for up to 10 Global Cache Service Processes.
    The number of LMSn processes varies depending on the amount of
    messaging traffic among nodes in the cluster. The LMSn processes
    handle the acquisition interrupt and blocking interrupt requests
    from the remote instances for Global Cache Service resources.
    For cross-instance consistent read requests, LMSn creates a
    consistent read version of the block and sends it to the
    requesting instance. LMSn also controls the flow of messages to
    remote instances.

    LMD - The Global Enqueue Service Daemon (LMD) is the lock agent process
    that manages enqueue manager service requests for Global Cache
    Service enqueues to control access to global enqueues and
    resources . The LMD process also handles deadlock detection and
    remote enqueue requests . Remote enqueue requests are requests
    originating from another instance .


    LCK - The LCK process has been simplified . Much of the LCK
    functionality in previous products is now handled by Global
    Cache Service Processes (LMSn). In Real Application Clusters,
    there is just one remaining LCK process . It is used for
    non-Cache Fusion operations such as instance requests and
    cross-instance call operations.

    BSP - No longer exist in 9i - Blocks shipping is carried out by GCS
    layers provided by LMSn

    We point you to [1] Sect.1 for further informations regarding OSD layers
    in linux (WD,CM and NM) that are still valid here .

    2) Architecture
    ------------

    2.1) Two-node / Two-instance RAC Archtecture
    ---------------------------------------

    Shared Disks,
    Database on RAW
    partitions
    / \
    / \
    Node 1 Node 2
    --------------- ---------------
    | SGA Instance A| |SGA Instance B |
    |---------------| |---------------|
    | |pmon|__ | | |pmon|___ |
    | |smon |lgwr|| | |smon |lgwr||
    | ---|dbwr| | | ---|dbwr| |
    | ---- | | ---- |
    | ------------| |------------ |
    | |lmon lms0 | |lmon lms0 | |
    lipc | | lmd | nipc | lmd | | lipc
    ------- GCS/GES <=================> GCS/GES -------
    | | |------------| |------------| | |
    | | | CM | | CM | | |
    +------> <===================> <------+
    | | |___________ | | ___________| | |
    | | | NM | | NM | | |
    +------> <===================> <------+
    | | |___________ | |____________| | |
    ------> WDT | | WDT <------
    | | | | | |
    --------------- ---------------
    |////kernel/////| |////kernel/////|
    --------------- ---------------

    '-' lipc : local inter-process communication
    '=' nipc : networked inter-process communication (via tcp/ip)

    Note : Typically the shared disks box is wired via SCSI or Optical
    Fiber to the main boards, this introduce compatibility issues related
    to the drivers used and to the controllers used . Please refer to HW
    certification matrices for that.

    2.2) Single-Node / Two-instance RAC Architecture
    -------------------------------------------

    Local Disk,
    Database on FILE SYSTEM

    |
    single Node |
    ---------------------------------------------
    | SGA Instance A| |SGA Instance B |
    |--------------- ---------------|
    | |pmon|__ |pmon|___ |
    | |smon |lgwr| |smon |lgwr||
    | ---|dbwr| ---|dbwr| |
    | ---- (localhost) ---- |
    | __________ nipc __________ |
    | | GCS/GES | <=========> | GCS/GES | |
    | | lmon | | lmon | |
    | | lmd lms0 | | lmd lms0 | |
    | |__________| |__________| |
    | ____|________________________|_____ |
    | | | | | |
    | | +--> CM svcs <--+ | |
    | |____|________________________|_____| |
    | | | | | |
    | | +--> NM svcs <--+ | |
    | |____|________________________|_____| |
    | | | | | |
    | | --> WDT svcs <-- lipc | |
    ---------------------------------------------
    |//////////////// kernel ////////////////////|
    ---------------------------------------------

    '-' lipc : local inter-process communication
    '=' nipc : networked inter-process communication (via tcp/ip)

    Note : this kind of configuration is not so exposed to
    compatibility issues related to various SCSI controller and
    drivers that may limit the implementations .

    Note : we will use normal ext2 file system, see below for
    further details (3.2) .



    3) Kernel and System Configuration
    --------------------------------
    we tested on Dell latitude pIII 500Mhz - 12Gb IDE - 256MB ram - CDROM)


    3.1) Bill of Materials [3]
    -------------------

    Intel x86 X-windowed and networked box
    SuSE 7.2,Red Hat 7.1 distributions
    Related Linux kernel source 2.4 rpms (kernel-2.4.XX-YY.i386.rpm)
    Oracle 9.0.1.0. for Linux Intel 32bit

    Operating System Linux : kernel 2.4
    Operating System Patches: Patch 4 or later
    Raw should be included in above distr.
    System Libraries : GNU Lib C 2.2,make-3.79,binutils-2.10

    Disk Space Requirements : ~2GB for an average install, Over 2.5GB for
    Default installation .


    3.2) - Prepare your disk for RAW partitions - (simplyfied)
    --------------------------------------------------------
    In [1] we showed that every database file was positioned on a dedicated
    RawDevice (RD) , and this common in a multinode RAC environments to
    override dangerous unsyncronized File System caches . In a single-node
    configuration we can remove the RD constraint because FS caches
    coerency issues does not arise us when the FS caches involved is
    just one .


    3.2.1) Cluster Manger RAW Partition can be on FS
    -----------------------------------------
    Node Monitor needs a raw partition to use as syncronization/voting aera
    among nodes partecipating the cluster . This partition can be a file in
    force of the single node hipotesys and the size should be at least :

    4 + [(the number of nodes in the cluster) * 4] KB [1]

    We will use 1M for our scopes (see 5.2)


    3.2.2) GSD Partition (optional)
    ------------------------
    If you plan to use 'SRVCTL' utility [4] then you will need a RawDevice
    partition of about 100M to let GSD (Global Services Daemon) work [5].
    Infact this daemon checks (with an iostat primitive) that the file
    descriptor points really to a RD .

    infact

    in : $ORACLE_HOME/srvm/config
    there is : 'srvConfig.loc'
    that contains : '/u01/ora/oradata/RAC_Cluster_Configuration_File'
    that should point to your RD .

    if you issue for example :

    % gsd
    % srvconfig -version

    will raise ->

    "oracle.ops.mgmt.rawdevice.RawDeviceException: PRKR-1020 :
    reading of version information failed,
    /u01/ora/oradata/RAC_Cluster_Configuration_File
    is not a raw device file, No such file or directory"

    To avoid this you must provide a true Raw Partition .



    3.3) Prepare your kernel for RAC single node
    ----------------------------------------
    In [1] we discussed in this section how to build the kernel and which
    parameters define in order to allow OPS work . Here we will remove some
    constraints in order to let your RAC environment be more usable for the
    scopes of this document .

    - we assume that you have kernel sources in place and you
    are familiar with linux kernel setup (see [1] Sect-3.3 for hints)

    3.3.1) KERNEL PARAMETERS relevant to RAC single-node
    ---------------------------------------------

    'Character Devices' -> 'Watchdog Cards' :

    CONFIG_WATCHDOG
    'Watchdog Timer Support' = 'y'

    CONFIG_SOFT_WATCHDOG
    'Software Watchdog' = 'm'

    CONFIG_WATCHDOG_NOWAYOUT
    'Disable watchdog shutdown on close' = 'n' (*)

    -----------------
    *Important Note* :
    -----------------
    RAC multinode needs 'y' [1] , infacts, the default watchdog
    behaviour (which you get if you say N here) is to stop the
    timer if the process managing it closes the file /dev/watchdog.
    It's always remotely possible that this process might get
    killed. If you say Y here, the watchdog cannot be stopped once
    it has been started . In our single-node environment is not
    unfrequent that 'watchdogd' hangs when OCM crash. In these
    cases we have to stop and restart, for this reason we use 'n'
    in order to avoid us to be forced to bounce the entire box when
    'watchdogd' stucks .


    3.3.2) Hack softdog.c
    --------------
    The software watchdog 'softdog' that we use in our environment have a
    pourpose in life : reboot the node when one of the registered processes
    forget to 'ping' watchdogd . This is reasonable multi-node RAC, because
    prevents that the entire cluster hangs due to fact that a single remote
    key processes (LMON..) does not respond to posts because is dead or is
    spinning . It is preferrable with linux port, to loose the node and
    reconfigure the cluster and let the system proceede .

    This happens for example, in case of an instance failure. The processes
    belonging to that RAC instance typically will not de-register correctly
    in case of a crash and softdog will trigger the node reboot.In our case,
    this is not so good, because we have more than one instance on the same
    node and we expect that we can continue with the surviving ones. For
    this reason we will modify the source code of the softdog, 'softdog.c',
    in order to skip the reboot (*)

    Currently, we observed that in certain types of sudden crashes we loose
    also OCM and conseguently the other instances goes down , this is
    normal and is tolerated within the limits of this special environment .


    (become root, we use kernel label '2.4.7', adapt to your one)

    % cd /usr/src/linux-2.4.7/drivers/char
    % cp softdog.c softdog.c.orig #-- make a backup copy


    modify softdog.c in this way, adding the line at the
    beginning of the file :

    #define ONLY_TESTING

    % cd /usr/src/linux-2.4.7
    % make modules


    wait until all modules are built (typically unless you performed
    a 'clean' all obj should be already in place and only our softdog
    will be rebuilt . Check that the module was rebuilt :

    % ls -l /usr/src/linux-2.4.7/drivers/char/softdog.o

    check if softdog is already loaded, shutdown all his clients and
    unload it :

    % lsmod

    Module Size Used by
    ...
    softdog 1504 1 <--- in this case YES
    autofs 10944 1 (autoclean)
    serial_cb 1632 1
    cb_enabler 2688 2 [serial_cb]
    ...

    % rmmod softdog

    substite the newly one :

    % cd /lib/modules/2.4.7/kernel/drivers/char
    % cp softdog.o softdog.o.orig #-- back it up
    % cp /usr/src/linux-2.4.7/drivers/char/softdog.o .

    now you can 'insmod' it again


    -----------------
    *Important Note* :
    -----------------
    RAC on Linux does not allow the system to continue upon a key
    process failure and the softdog.c hack has to be intended
    finalyzed to scopes and limitations of this article .

    >>Modifying the sofdog code when using RAC is not supported <<

    3.3.3) Kernel memory requirements
    --------------------------
    Oracle 9i naturally is more memory consumer than previously
    releases and you have to instruct linux kernel to allow bigger
    memory mappings.

    The followings are a suitable configuration, merge it in your
    /etc/rc.d/rc.local :


    #--- Oracle 9i needed : rc.local PART 1/2
    cd /proc/sys/kernel

    cat sem
    export SEMMSL=200
    export SEMMNS=512
    export SEMOPM=200
    export SEMMNI=200

    echo $SEMMSL $SEMMNS $SEMOPM $SEMMNI > sem

    cat shmmax shmmni shmmni shmseg
    export SHMMAX=2147483648
    export SHMMIN=1
    export SHMMNI=100
    export SHMSEG=4096

    echo $SHMMAX > shmmax
    echo $SHMMIN > shmmin
    echo $SHMMNI > shmmni
    echo $SHMSEG > shmseg

    cat shmmax shmmni shmmni shmseg
    #--- End of rc.local PART 1/2



    4) install Oracle software
    -----------------------
    Simply install the software as usual taking in care of explicitly select :

    'Oracle Real Application Clusters 9.0.1.0.0'

    from the software list in addition to the other options you will need .

    This will install :

    Oracle Cluster Manager 2.0.0.10.0 (in $ORACLE_HOME/oracm)
    Oracle RAC Configuration Tool

    5) Setup Oracle Cluster Manager components
    ---------------------------------------

    5.1) load the softdog module in your kernel at startup
    -------------------------------------------------
    appending to your system startup file the :

    #--- Oracle 9i needed : rc.local PART 2/2
    # (this command must be done by root)

    /sbin/insmod softdog soft_margin=60

    #--- End of rc.local PART 2/2


    5.2) Create the NM File
    ------------------
    we will use a 1M file that is enough

    % dd if=/dev/zero of=/u01/ora/oradata/RAC_Node_Monitor_file \
    bs=1024 count=1024

    1024+0 records in
    1024+0 records out

    5.3) Configure Oracle Cluster Manger (OCM)
    -------------------------------------
    like for OPS, in RAC for linux the OSD layer is developped by Oracle and
    is located under $ORACLE_HOME/oracm . Here you will find :

    $ORACLE_HOME/oracm/
    admin/ -> CM configuration files resides :
    nmcfg.ora -> Node Monitor conf. file
    ocmargs.ora -> OCM startup parameters

    bin/ -> the modules :
    ocmstart.sh -> OCM startup script
    oracm -> CM binary
    oranm -> NM binary
    watchdogd -> WD binary

    rawio rdevtest -> troubleshooting tools
    wdttest1 wdttest2

    log/ -> where the logs will go :
    cm.log cm.out -> CM log stream and stdout
    nm.log nm.out -> NM log stream and stdout
    ocmstart.ts -> ocmstart.sh lock file
    wdd.log -> WD out

    configure the NM :

    edit 'nmcfg.ora' and substitute the 'DefinedNodes' with
    your `hostname` , and 'CmDiskFile' with the file defined
    above (5.2)

    DefinedNodes=cclinux
    CmDiskFile=/u01/ora/oradata/RAC_Node_Monitor_file

    Note : the node name should be returned by 'hostname' and
    resolved in /etc/hosts (if you're using DHCP you may set the
    /etc/hosts alias for 127.0.0.1 to the output of `hostname`).
    This will prevent unnecessary IP roundtrips .


    configure OCM :

    edit 'ocmargs.ora' and check the following defaults

    watchdogd -g dba
    oranm
    oracm
    norestart 1800

    if you take a look to 'ocmstart.sh' you will understand the
    meaning of each OCM parameters


    5.4) Troubleshooting Oracle Cluster Manger (OCM)
    -------------------------------------------
    may happen that OCM won't start, the reasons should be investigated
    analyzing the log files of NM,CM,WD .

    5.4.1) Check filesystem rights
    -----------------------
    When starting OCM , 'ocmstart.sh' may return something like :

    oranm: cnmrequest.c:138: ConnectListener: Assertion `0' failed

    the 'nm.log' shows :
    ...
    WARNING | ConnectListener(): failed to remove /tmp/.oranmsock errno =1
    MESSAGE | failed to bind at /tmp/.oranmsock errno=98
    ...

    infact in this case :

    % ls -l /tmp/.ora*sock*
    srwxrwxrwx 1 root bin 0 Dec 3 11:23 /tmp/.oracmsock
    srwxr-xr-x 1 root bin 0 Dec 3 11:23 /tmp/.oranmsock

    (stop all OCM and remove them , they will be re-created)

    If you start OCM with 'oracle' account, then verify that you
    have some FS rights to do that .

    /tmp/.watchdog/* - directory where resides the sockets used
    by 'oracm','oranm', 'lmon' <-> 'watchdogd'
    in the form cl_sock__XXXX
    /tmp/.oranmsock - socket for NM
    /tmp/.oracmsock - socket for CM

    /u01/ora/product/9.0.1/oracm/log/ocmstart.ts - Lock semaphore

    5.4.2) Oracle does not start
    ---------------------
    typically when facing errors like this during the startup

    SQL> startup
    CM:CmThreadAttach: socket not connected (error=111)

    you should check that all modules of OCM are running, if not, or
    only a part of OCM is running , stop all OCM modules and restart
    them again


    5.4.3) OCM did not restart
    -------------------
    if you experience errors like :

    ocmstart.sh: Error: Restart is too frequent
    Info: Check the system configuration and fix the problem.
    Info: After you fixed the problem, remove the timestamp file
    Info: "/u01/ora/product/9.0.1/oracm/log/ocmstart.ts"

    and OCM is not running , you need to remove the semaphore file
    indicated .

    % rm /u01/ora/product/9.0.1/oracm/log/ocmstart.ts

    5.4.4) is OCM running ?
    -----------------
    a quick way to check if OCM is running is to check if you can see a
    similar processes tree:

    watchdogd -g dba
    oranm
    \_ oranm
    \_ oranm
    \_ oranm
    \_ oranm
    \_ oranm
    \_ oranm
    oracm
    \_ oracm
    \_ oracm

    5.4.5) how to stop OCM
    ---------------
    basically you have to kill all the above OCM related processes, after
    you determe that there are not any RAC instance is running .

    %
    %
    %


    6) Clustered database setup
    -----------------------
    At this point our system is ready to start a couple of instances on the
    same database, but before that, we have to create the database. A database
    that is intended to be opened in cluster mode is quite the same that one
    in exlusive mode, except for some main issues .

    - SPFILE.ora : In a clustered database we will have an specific section
    for each instance that will mount the database . With 9i, we introduced
    the SPFILE (Server Parameter FILE) that store all the instance parameters.
    SPFILE's command syntax is CLUSTER-Awared and we can specify all our needs
    in a single file . With 9i, in a multi-nodes RAC configuration, SPFILE can
    be efficiently pointed to a shared RD partition simplyfiing the setup and
    avoiding parameter misconfiguration .

    - Redo Logs : In a clustered database each instance must have his own Redo
    Logs, so we will talk of 'Redo Log Threads' (RLT). RLT are numbered from 1
    to n and normally we have only one thread of Redo Logs and this is the RLT
    #1. In RAC each starting instance will use exclusively the RLT number 'n'
    where 'n' is equal to the 'thread=n' parameter specific for that istance .

    - Rollback Segments : Each instance must acquire at least one private RBS,
    wich one is controlled with instance specific 'rollback_segment=(....)'
    parameter. This is the pre-9i way to define the undo space for OPS , 9i
    introduced the concept of Self Managed UNDO Space, through the concept of
    UNDO tablespace built in the database. In a RAC environment, you should
    setup an UNDO tablespace for every instance you plan to join the database.


    6.1) Create RAC database
    -------------------
    DataBase Configuration Assistan (DBCA) can be used to build the scripts
    that define our database . DBCA is a RAC-Awared utility, in therms that it
    can drive you through the creation of a clustered database but to do that
    require it an extra setup not covered by this document (see GSD,srvconfig,
    srvctl .. [4],[5]) .

    We prefer to create a normal database as we are used, then 'transform' it
    in a 'clusterizable' one . In this way we think that we will understand
    better some concepts that can be helpfull if using more sophisticated
    tools like DBCA in the future .


    6.1.1) Database on File System , easyer to setup on FS
    ---------------------------------------------
    In force of above hipotesys (3.2) we will use normal File System
    datafiles (*) simplyfing the environment we are going to setup

    DataBase 'NINEI'
    ----------------
    Create it using your preferred script/tool ...

    PATH |Description
    ----------------------------------------|---------------
    $ORACLE_BASE/oradata/NINEI/
    control01.ctl control02.ctl <- Controlfiles
    +) redo11.log +) redo12.log <- Thread#1 RLogs
    +) redo21.log +) redo22.log <- Thread#2 RLogs
    system01.dbf
    temp01.dbf
    tools01.dbf
    users01.dbf
    undotbs01.dbf <- UNDO TS Instance 1 (20M)
    +) ora_undotbs__y0hrj0og.dbf <- UNDO TS Instance 2 (20M)
    (autogenerated)

    $ORACLE_BASE/oradata/
    RAC_Node_Monitor_file <- Node Monitor


    Note : the '+' marked files are created by the next steps .

    Note : as you can see is quite like a normal test database and
    the respective sizes are also nothing special . In following
    sections we will assume that you start from a standard database
    created with DBCA using OFA statements.


    -----------------
    *Important Note* :
    -----------------
    DO NOT USE File System based RAC databases in a multi-node
    environment unless it is explicity certified and supported (i.e.
    Tru64 RAC or OpenVMS RAC ..)



    6.1.2) create RAC related catalogs
    ---------------------------

    % cd $ORACLE_HOME/rdbms/admin
    % sqlplus "/ as sysdba"

    @catparr.sql


    6.1.3) Create extra RLog Threads
    -------------------------
    once the database is started exclusive the first time, we can
    create the RLT for both the instances :

    %sqlplus "/ as sysdba"

    ALTER DATABASE ADD LOGFILE THREAD 1 GROUP 11
    '/u01/ora/oradata/NINEI/redo11.log size' 10M ;
    ALTER DATABASE ADD LOGFILE THREAD 1 GROUP 12
    '/u01/ora/oradata/NINEI/redo12.log size' 10M ;
    ALTER DATABASE ADD LOGFILE THREAD 2 GROUP 21
    '/u01/ora/oradata/NINEI/redo21.log size' 10M ;
    ALTER DATABASE ADD LOGFILE THREAD 2 GROUP 22
    '/u01/ora/oradata/NINEI/redo22.log size' 10M ;
    ALTER DATABASE ENABLE THREAD 2;

    (optionally use the 'reuse' clause)

    6.1.4) Create extra UNDO tablespaces
    -----------------------------
    every 9i RAC instance needs his own UNDO tablespace when using
    undo_management=AUTO :

    %sqlplus "/ as sysdba"

    create UNDO tablespace UNDOTBS_B datafile size 20M;

    6.2) setup for use SPFILE
    --------------------
    Assuming that you created a database named 'NINEI' and an instance
    named 'NINEI' for normal non-clustered operations, we are going to
    create the SPFILE that will used by all the instances when mounting
    clustered database .

    normally is (or check that it is) :

    % cd $ORACLE_HOME/dbs
    % ls -l initNINEI.ora

    initNINEI.ora -> /u01/ora/admin/NINEI/pfile/initNINEI.ora

    where 'initNINEI.ora' contains all parameters, now we create a new SPFILE
    to use with RAC starting from this file

    % sqlplus "/ as sysdba"

    create spfile='/u01/ora/admin/NINEI/pfile/spfileNINEI.ora.RAC'
    from pfile='u01/ora/admin/NINEI/pfile/initNINEI.ora';

    now save a backup copy of you initNINEI.ora and edit . modify it
    in order to contain only the following line :

    spfile=/u01/ora/admin/NINEI/pfile/spfileNINEI.ora.RAC

    pointing to our new SPFILE . Verify that shutting down the instance and
    then starting up all is working with SPFILE .


    6.3)Configure SPFILE for RAC
    ------------------------
    the following steps assume that we will start the clustered database from
    two instances named 'RAC_A' and 'RAC_B', errors like :

    'ORA-32010: cannot find entry to delete in SPFILE '

    are expected and can be ignored . Note also that the order to run
    these statements is relevant ..

    Key
    Point | RAC Parameter
    ------|-----------------------------------------------------
    (1)
    alter system reset cluster_database scope=spfile sid='*';
    alter system set cluster_database=TRUE scope=spfile sid='RAC_A';
    alter system set cluster_database=TRUE scope=spfile sid='RAC_B';

    (2)
    alter system reset instance_name scope=spfile sid='*';
    alter system set instance_name=RAC_A scope=spfile sid='RAC_A';
    alter system set instance_name=RAC_B scope=spfile sid='RAC_B';

    (3)
    alter system reset undo_tablespace scope=spfile sid='*';
    alter system set undo_tablespace=UNDOTBS scope=spfile sid='RAC_A';
    alter system set undo_tablespace=UNDOTBS_B scope=spfile sid='RAC_B'

    (4)
    alter system reset instance_number scope=spfile sid='*';
    alter system set instance_number=1 scope=spfile sid='RAC_A';
    alter system set instance_number=2 scope=spfile sid='RAC_B';

    (5)
    alter system reset thread scope=spfile sid='*';
    alter system set thread=1 scope=spfile sid='RAC_A';
    alter system set thread=2 scope=spfile sid='RAC_B';

    (6)
    alter system reset cluster_database_instances scope=spfile sid='*';
    alter system set cluster_database_instances=2 scope=spfile sid='*';

    (7) [optional if not in ArchiveLogMode]
    alter system reset log_archive_dest_1 scope=spfile sid='*';
    alter system reset log_archive_dest scope=spfile sid='*';
    alter system set
    log_archive_dest_1='location=/u01/ora/oradata/NINEI/logA'
    scope=spfile sid='RAC_A';
    alter system set
    log_archive_dest_1='location=/u01/ora/oradata/NINEI/logB'
    scope=spfile sid='RAC_B';
    alter system reset log_archive_format scope=spfile sid='*';
    alter system set
    log_archive_format='log_%t_%s' scope=spfile sid='*';

    (8) [optional if not using TAF features]

    alter system reset service_names scope=spfile sid='*';
    alter system set service_names='NINEI,SERVICE_A'
    scope=spfile sid='RAC_A';
    alter system set service_names='NINEI,SERVICE_B'
    scope=spfile sid='RAC_B';


    Key Point Considerations :
    ------------------------
    (1) We explicity set 'CLUSTER_DATABASE=TRUE' for each instance
    and not with the 'sid=*' qualifier in order to allow us to start
    the 'NINEI' database in exlusive mode if needed . It will be
    enough to switch to the orginal 'NINEI' instance environment
    used for the creation .

    (2) Usefull for service registration and OEM .

    (3) Every instance have his own UNDO tablespace

    (4),(5) Obvious

    (6) In 9i RAC, this parameter is used to dimensionate the inner
    kernel structures that lay under GCS/GES, it's used for DLM Fast
    Reconfiguration algorithms (Lazy Remastering). It should be
    setted to the number of potential instances joining the
    clustered database .

    (7) RAC instances normally keep different archive log destinations

    (8) used for TAF configuration & tests




    7) Startup both the instances in shared mode
    -----------------------------------------
    Assuming that your oracle is linked with RAC enabled, we perform the
    following task from the 'oracle' unix account .

    7.1) Start OCM
    ---------
    login with you 'oracle' account


    check that softdog is in place (5.1)

    % /usr/sbin/lsmod | grep softdog

    start OCM with the new provided script

    % cd $ORACLE_HOME/oracm/bin
    % ./ocmstart.sh



    7.2) setup unix environment
    -------------------------
    in order to have the ORACLE_HOME pointing to the 9.0.1.0.0 code path
    and two distict ORACLE_SID :

    Instance RAC_A Instance RAC_B
    -------------- --------------

    ORACLE_SID=RAC_A ORACLE_SID=RAC_B
    ORACLE_BASE=/u01/ora ORACLE_BASE=/u01/ora
    ORACLE_HOME=/u01/ora/product/9.0.1 ORACLE_HOME=/u01/ora/product/9.0.1
    ORACLE_TERM=vt100 ORACLE_TERM=vt100
    PATH=$PATH:$ORACLE_HOME/bin: PATH=$PATH:$ORACLE_HOME/bin:
    PATH=$PATH:$ORACLE_HOME/oracm/bin PATH=$PATH:$ORACLE_HOME/oracm/bin
    LD_LIBRARY_PATH=$ORACLE_HOME/lib LD_LIBRARY_PATH=$ORACLE_HOME/lib
    export ORACLE_BASE export ORACLE_BASE
    export ORACLE_HOME export ORACLE_HOME
    export ORACLE_TERM export ORACLE_TERM
    export ORACLE_SID export ORACLE_SID
    export PATH export PATH
    export LD_LIBRARY_PATH export LD_LIBRARY_PATH

    7.3) startup the instances
    ---------------------
    The usual symbolic links to init files should already be created in
    $ORACLE_HOME/dbs .

    initRAC_A.ora -> /u01/ora/admin/NINEI/pfile/initNINEI.ora
    initRAC_B.ora -> /u01/ora/admin/NINEI/pfile/initNINEI.ora

    Note : we point to the same init.ora that contains the pointing
    to the same shared SPFILE .


    % sqlplus "/ as sysdba" % sqlplus "/ as sysdba"
    startup startup
    ... ...

    if all worked ok, both the istances will startup, monitor
    'alertRAC_A.log' , 'alertRAC_B.log' for errors

    the process view should show the follwing process runnings :


    ora_pmon_RAC_A ora_pmon_RAC_B
    ora_diag_RAC_A ora_diag_RAC_B
    ora_lmon_RAC_A ora_lmon_RAC_B
    ora_lmd0_RAC_A ora_lmd0_RAC_B
    ora_lms0_RAC_A ora_lms0_RAC_B
    ora_lms1_RAC_A ora_lms1_RAC_B
    ora_dbw0_RAC_A ora_dbw0_RAC_B
    ora_lgwr_RAC_A ora_lgwr_RAC_B
    ora_ckpt_RAC_A ora_ckpt_RAC_B
    ora_smon_RAC_A ora_smon_RAC_B
    ora_reco_RAC_A ora_reco_RAC_B
    ora_arc0_RAC_A ora_arc0_RAC_B
    ora_lck0_RAC_A ora_lck0_RAC_B


    8) Switching between RAC and NON-RAC
    ---------------------------------
    You can switch back your oracle to a NON-RAC mode relinking the executable.
    If you use the default 'NINEI' SID name you can point to the same SPFILE .

    8.1) Activating RAC feature
    ----------------------
    If you want to open your database in SHARED MODE (CLUSTER_DATABASE=TRUE)
    you MUST have oracle RAC-ENABLED

    % cd $ORACLE_HOME/rdbms/lib
    % make -f ins_rdbms.mk rac_on
    % make -f ins_rdbms.mk ioracle

    you can however open a database with CLUSTER_DATABASE=FALSE using a RAC
    ENABLED oracle .

    8.2) De-activacting RAC feature
    --------------------------
    If you want to open any database without OCM running you MUST have oracle
    RAC-DISABLED .

    % cd $ORACLE_HOME/rdbms/lib
    % make -f ins_rdbms.mk rac_off
    % make -f ins_rdbms.mk ioracle

    8.3) Check if RAC is enabled or not
    ------------------------------
    In linux (only) you can use the follwing statement in order to chech if
    an Oracle RAC specific library is linked against the current executable

    % ldd $ORACLE_HOME/bin/oracle | grep libwddapi.so

    IF <1 row> IS RETURNED
    THEN your oracle IS RAC-ENABLED
    ELSE your oracle IS NOT RAC-ENABLED




    9) Conclusions
    -----------
    The solution exposed here looses most of the key features of a RAC
    solution that depends on multiple nodes to scale-up application
    performances and to be Highly Available eliminating single points of
    failures such instances and nodes .

    This document should be intended principally for didactic scopes and used
    by peoples that are testing the RAC technology and need to setup a
    quick solution for simulation, test and developement purposes .


    The 'RAC-in-a-box' solution can be suggested also for :

    - estimate scaling factor curve for an application with RAC
    - scale up on a single node with multiple SGAs in case of
    very high latch contention issues
    - more than 2 clustered instances, we tested with up to 6 RAC
    instances on a laptop



    -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --

    RELATED DOCUMENTS
    -----------------


    [1] Note:136077.1 Setting up Oracle Parallel Server
    environment on Linux - Single node

    [2] Oracle9i RAC - Concepts (9.0.1)

    [3] Oracle9i Release Notes (9.0.1) for Linux Intel

    [4] Note:159371.1 SRVCTL utility in Oracle9I RAC

    [5] Oracle9i RAC - Installation and Configuration


Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  


Click Here to Expand Forum to Full Width