Oracle Labs by Yuri Khazin, Oracle DBA

Virtual Oracle RAC. Task 10: Oracle installation in RAC nodes – part 1


Link back: This guide is a part of the Virtual Oracle RAC project, the index to the whole project is here.

Taking care of memory

As mentioned before, now comes the time to increase the virtual machines memory to 1024MB. Take this chance to run a backup of the whole project while guests are shut down, then change Base Memory settings on both nodes. Start all the machines again.

Getting an X Server running on the PC

There are few small things to take care of before we launch Oracle Installer. Namely, we need an X Server on our PC. This can be any server of your choosing, such as Cygwin/X, Xming or other. I will be using Cygwin/X here. Make sure your X Server includes xhost utility, sometimes it is not installed by default. This is how to locate the xhost utility during instalation:

Have your X Server installed and started on your PC. If using Cygwin, run in CMD window (or via an icon):

C:\bin\cygwin2\bin\run.exe /usr/bin/bash.exe -l -c /usr/bin/startxwin.exe

This will usually start X server on display 0.0 and bring up an xterm window. If xterm window did not appear have it started manually. In this xterm window give an access control command:

$ xhost +10.10.1.11

$ xhost +10.10.1.12

This xhost command has enabled the obdn1 host to use your X Server to display x clients. I think I have just saved you a few days of frustration. The access control matters of X Server are not covered clearly in documentation and articles would suggest using X11 forwarding in SSH. The X11 forwarding in SSH will indeed take care of all the neccessary settings automaticatlly, however this setup won’t be usable by Oracle Installer. Therefore you should check that the SSH sessions you run from your PC to the RAC nodes are not using the X11 forwarding.

So, if you are using PuTTY make sure the X11 forwarding is not selected for the RAC node sessions:

As user "oracle", from the odbn1 node, from that same SSH session you have enabled SSH no-password connections (go back to refresh your memory). Ff you have restarted servers or logged in again in SSH you will need to repeat steps with ssh-agent and ssh-add as described before (on odbn1 only).

$ export DISPLAY=10.10.1.1:0

Test that your session can run some X windows application and show it on your PC’s X Server:

$ xclock &

This should spawn an xclock application and show it on your PC via X Server. If this did not work you will have to troubleshoot your setup and proceed with Oracle installation only when resolved.

Notice: The IP address we supplied in DISPLAY variable is the virtual "router" on our host PC, since X Server expect connection from 10.10.1.11, we should "talk" to it on the same network.

Having endured so far we are now going to do more exciting things.

Configure OCFS2

Notice: OCFS2 package (RPM) installation instructions are in Task 7 , go back if you missed them. If you already cloned the node then do this step on both nodes.

OCFS2 will be configured to use private network i.e. hostnames as in etc/hosts:

10.10.1.11 odbn1-priv
10.10.1.12 odbn2-priv

Notice, that although we will be using the private network IP addresses we will still use hostnames as they appear on public network. I know, it is confusing. There would not be such a confusion with just one network (either private or public).

As user "root", on the odbn1 node, start a new SSH session and set the DISPLAY varaible:

# export DISPLAY=10.10.1.1:0

 

Now, bring up OCFS2 configuration tool to edit /etc/ocfs2/cluster.conf file:

# ocfs2console &

Select [ Cluster ] – [ Configure Nodes ]. This will start the OCFS2 Cluster Stack and bring up next dialog ("Node Configuration")

There will be a pop-up message:

Dismiss it and proceed to "Node Configuration". Add two RAC nodes as below:

When you click "Apply" the screen changes to node status and node numbers:

Exit the console and repeat the same process on the other node of the cluster supplying same exact data in the same order.

The /etc/ocfs2/cluster.conf file will look like this after configuration is done (do not just copy this file over to other node):

O2CB Cluster Service

O2CB is the OCFS2’s cluster stack of services that are listed below:

  • NM: Node Manager that keep track of all the nodes in the cluster.conf
  • HB: Heart beat service that issues up/down notifications when nodes join or leave the cluster
  • TCP: Handles communication between the nodes
  • DLM: Distributed lock manager that keeps track of all locks, its owners and status
  • CONFIGFS: User space driven configuration file system mounted at /config
  • DLMFS: User space interface to the kernel space DLM

All of the listed services are included in the single service o2cb (/etc/init.d/o2cb).

Now issue following commands to have O2CB start on boot up and have other needed attributes (on both nodes):

# /etc/init.d/o2cb offline ocfs2
# /etc/init.d/o2cb unload
# /etc/init.d/o2cb configure

Below shown non-default responses to the configuration questions (marked by arrows):

Repeat the same steps on the other node.

Issue this command to review the service status so far:

# /etc/init.d/o2cb status

Make sure that o2cb serivce will be started at proper run level:

# chkconfig –list o2cb

Levels 3 through 5 should be "on":

Format and mount the OCFS2 Filesystem

This task is to be done from one node only

# mkfs.ocfs2 -b 4K -C 32K -N 4 -L oracrsfiles /dev/iscsi/crs/part1

The following output is produced:

Now mount the newly formatted partition (that was labeled "oracrsfiles") under directory /u02:

# mount -t ocfs2 -o datavolume,nointr -L "oracrsfiles" /u02

Repeat the mount command on the other node. Verify that the file system is mounted correctly:

# mount

Repeat verification on the other node. Your output should be similar to the one below (newly mounted file system is highlighted):

Make these changes permanent by adding the following line to the /etc/fstab file on both RAC nodes:

LABEL=oracrsfiles /u02 ocfs2 _netdev,datavolume,nointr 0 0

This is how it will look like:

Adjust ownership and permissions on the new filesystem (from one node only):

# chown oracle:oinstall /u02
# chmod 775 /u02

Create directories to contain Oracle Clusterware shared files (also from one node only):

# mkdir -p /u02/oradata/racdb
# chown -R oracle:oinstall /u02/oradata
# chmod -R 775 /u02/oradata

Reboot both nodes. Check if our setup can survive the reboot. Use "mount" command to see if the new file system is still there and is mounted in the proper place. You can also use o2cb status check to make sure the serive is functioning properly (from both nodes):

# /etc/init.d/o2cb status

 

Configure ASMLib 2.0

Notice: ASMLib package (RPM) was added in Task 7 , go back if you missed this step. If you have already cloned the node then do this step on both nodes.

The configuration task is to be done on both nodes:

# /etc/init.d/oracleasm configure

The following picture shows your input marked by red arrows:

 

Create ASM disks

We are creating the ASM disks from one node only. Run following commands:

# /etc/init.d/oracleasm createdisk VOL1 /dev/iscsi/asm1/part1
# /etc/init.d/oracleasm createdisk VOL2 /dev/iscsi/asm2/part1
# /etc/init.d/oracleasm createdisk VOL3 /dev/iscsi/asm3/part1
# /etc/init.d/oracleasm createdisk VOL4 /dev/iscsi/asm4/part1

The commands above have created ASM disks in iSCSI targets that are mapped to local devices:

The newly created disks can be listed by using this command:

# /etc/init.d/oracleasm listdisks

On the other node we only need to do a scandisk to recognize the new volumes and then list them:

# /etc/init.d/oracleasm scandisks
# /etc/init.d/oracleasm listdisks

 

Prepare Oracle software media

Download the following from http://www.oracle.com/technetwork/database/10201linuxsoft-097986.html:

Software Title File Name

Oracle Clusterware Release 2 (10.2.0.1.0)

10201_clusterware_linux32.zip

Oracle Database 10g Release 2 (10.2.0.1.0)

10201_database_linux32.zip

Oracle Database 10g Companion CD Release 2 (10.2.0.1.0)

10201_companion_linux32.zip
   

Now there is two ways to proceed. You can either:

  • copy the archives (with sftp) onto odbn1 node as user "oracle", into some directory
  • unzip the archives and re-pack them into an ISO image, then mount the image in VB guest

I will be using the second option as not to inflate the file system of the virtual machine (after all, it is a one time install).

So, have the archives unzipped into separate directories on your PC, then use a utility such as Infra Recorder to pack the software into a DVD ISO image, just like in the picture below:

Name this image "rac32media.iso":

Add newly created image to Virtual Media Manager and then mount it in both VB machines. Then mount it in the nodes like this:

# mount -r /dev/cdrom /media/cdrom
# cd /media/cdrom

Install cvuqdisk package

This package is needed later by Cluster Verification Utility (CVU). Issue the following commands (do it on both nodes):

# cd /media/cdrom/10201_clusterware_linux32/clusterware/rpm
# export CVUQDISK_GRP=oinstall
# rpm -iv cvuqdisk-1.0.1-1.rpm

Verify installation of the package:

# ls -l /usr/sbin/cvuqdisk

Install and run Cluster Verification Utility (CVU)

This utility will be run from odbn1 node as user "oracle". The SSH agent will have to be initiated (again) for this purpose.

Open another SSH session (we may want to use our "root" session later) and login as "oracle", then execute following commands:

$ exec /usr/bin/ssh-agent $SHELL
$ /usr/bin/ssh-add

Install the CVU utility:

Verifying Oracle Clusterware Requirements with CVU

Once all prerequisites for running the CVU utility have been met, we can now check that all pre-installation tasks for Oracle Clusterware are completed by executing the following command as the "oracle" UNIX user account from linux1:

$ mkdir -p /home/oracle/bin/cluvfy
$ cd /home/oracle/bin/cluvfy
$ cp /media/cdrom/10201_clusterware_linux32/clusterware/cluvfy/* .
$ mkdir -p jdk14
$ unzip jrepack.zip -d jdk14
$ export CV_HOME=/home/oracle/bin/cluvfy
$ export CV_JDKHOME=/home/oracle/bin/cluvfy/jdk14

Run pre-requisites check report

$ ./runcluvfy.sh stage -pre crsinst -n odbn1,odbn2 -verbose | tee cvu_report_1.txt

Review the CVU report (cvu_report_pre.txt). There are several errors like "Check: Package existence" that can be ignored.

There is a messages in connectivity section reading "Could not find a suitable set of interfaces for VIPs". This one can be ignored too, as it is caused by a known bug documented in Metalink Note 338924.1 (has to do with non routable ranges of IP addresses).

If you get any errors other than those in my report there might be a problem with your setup.

Connectivity errors must be resolved. For reference and troublesooting see Metalink ID 338924.1 and Forum

Run the Hardware and OS Setup check report

$ ./runcluvfy.sh stage -post hwos -n odbn1,odbn2 -verbose | tee cvu_report_2.txt

Review the report, it should say "Post-check for hardware and operating system setup was successful."

The utility checks connectivity again and then checks storage, proper report will look like this:

When reports are ok and all serious problems have been resolved we can move on.

Install Oracle Clusterware

The two shared files (file groups) used by Oracle Clusterware will be stored on the Oracle Cluster File System (OFCS2) we created earlier. The two shared Oracle Clusterware file groups are:

  • Oracle Cluster Registry (OCR)

File 1 : /u02/oradata/racdb/OCRFile
File 2 : /u02/oradata/racdb/OCRFile_mirror
Size : (2 * 100MB) = 200M

  • CRS Voting Disk

File 1 : /u02/oradata/racdb/CSSFile
File 2 : /u02/oradata/racdb/CSSFile_mirror1
File 3 : /u02/oradata/racdb/CSSFile_mirror2
Size : (3 * 20MB) = 60MB

Issue these commands as user "oracle":

$ cd ~
$ /media/cdrom/10201_clusterware_linux32/clusterware/runInstaller -ignoreSysPrereqs &

The Oracle Installer will appear, click "Next" as in the picture below:

Have Name and Path filled as shown, click Next:

In the next screen the installer runs various checks (the checks must be all right if our manual check reports were fine) and shows the final result, click Next:

The Cluster Configuration screen appears with one node filled out already, click on Add and supply information for the other node as shown so it looks like this:

Click "Next". The Network Interface configuration screen appears (it shows subnets and network adapter names). As you can see, both interfaces are shown as "Private". We need a public interface in this configuration. Select "eth0" and click "Edit".

Designate "eth0" as public interface. Click "Ok"

Our interfaces and their purpose look like this now:

Click "Next" and OCR location dialogue appears.

Supply following OCR locations in the next screen (leave Normal Redundancy selected):

Specify OCR Location: /u02/oradata/racdb/OCRFile
Specify OCR Mirror Location: /u02/oradata/racdb/OCRFile_mirror

Click Next:

Supply following locations of voting disks in the next screen (again, leave Normal Redundancy selected):

Voting Disk Location: /u02/oradata/racdb/CSSFile
Additional Voting Disk 1 Location: /u02/oradata/racdb/CSSFile_mirror1
Additional Voting Disk 2 Location: /u02/oradata/racdb/CSSFile_mirror2

Click Next:

Summary screen appears. Click Install:

Notice how, as the install progresses, the software is being copied over to the other node in the cluster

Notice: Sometimes one of the virtual machines may crash on memory related errors (running out of usable memory on the host PC).

If this happens on odbn2 it is safe to reboot it and wait till installer continues (say 5 minutes after reboot).

If this happens on odbn1 it is more complicated. You will need to:

  • cancel the installer
  • restart machines if they are down
  • delete subdirectories under of the /u01/app/crs and /u01/app/oracle
  • shut down all VM’s
  • free up host PC’s memory or reboot it
  • restart VM’s
  • restart the Installer
  • good luck

Upon completion of the remote copy operations (ssh and tar processes) the next screen shows up:

And pop up screen shows:

Now pay attention closely.

Important: Do not close this Oracle Installer dialog until instructed to do so.

Important: run orainstRoot.sh script on both nodes, one at a time.

Important: run /u01/app/crs/root.sh script on odbn1 and wait until it completes, then proceed with the same script on the odbn2.

Output of the root.sh script on odbn1:

The /var/log/messages shows this on the first node at the completion if the script:

Now proceed with the root.sh on odbn2 node (read a note further on). The output is here:

The error at the end (libpthread.so.0) is expected, here is the manual fix to this trouble.

Edit "/u01/app/crs/bin/vipca" file on both nodes adding the following statement after line 124:

unset LD_ASSUME_KERNEL

The snapshot below shows edited text with added "unset" command:

Next, edit "/u01/app/crs/bin/srvctl" file on both nodes adding same statement after line 167 so it will look like this:

To take care of this problem in future, when more software is installed into new Oracle Homes or patches applied to existsing software, the following files have to be re-edited:

<ORA_CRS_HOME>/bin/vipca
<ORA_CRS_HOME>/bin/srvctl
<RDBMS_HOME>/bin/srvctl
<ASM_HOME>/bin/srvctl

Since "vipca" utility failed, we will re-run it manually. Run vipca from odbn2 (still as root):

# export DISPLAY=10.10.1.1:0
# /u01/app/crs/bin/vipca

The vipca fails again with this cryptic message:

" [Error 0(Native: listNetInterfaces:[3])]"

The message has to do (again) with the range of IP addresses we selected for the interfaces.

Notice: If you have received another cryptic message [Exception in thread "main"] this simply means you forgot to set the DISPLAY variable.

This trouble has to be manually fixed.

Run following commands as root from odbn2 (notice that the IP addresses here are of the networks, rather than specific adapters):

# /u01/app/crs/bin/oifcfg setif -global eth0/10.10.2.0:public
# /u01/app/crs/bin/oifcfg setif -global eth1/10.10.1.0:cluster_interconnect

Let’s see now the resulting configuration of VIP network:

# /u01/app/crs/bin/oifcfg getif

So, let’s run vipca again (still on odbn2)

# /u01/app/crs/bin/vipca

The following screen pops up (click on Next):

Agree wih the next screen (eth0 is indeed the public interface):

As soon as we supply first IP alias name as "odbn1-vip" the rest of fields is magically filled out. Remove the domain name from the second alias. Click "Next".

Summary screen appears. Click "Finish".

The progress screen is shown:

When this is finished and you uncross your fingers, this screen shows. Click "Ok":

The summary appears:

Exit the vipca.

Now you can back to Clusterware installer dialog and dismiss the "Execute Configuration scripts" dialogue.

The installer continues

And … fails on Cluster Verification Utility:

We ignore this message and exit the installer.

Now let’s run a few verification commands on both nodes:

# /u01/app/crs/bin/olsnodes -n
# /u01/app/crs/bin/crs_stat -t -v
# /u01/app/crs/bin/crsctl check crs
# ls -l /etc/init.d/init.*

Here are the results (one node shown, the other must show identical results):

Now we are ready to install Oracle database software.

Next chapter.

Advertisements

2 Comments »

  1. Hi Yumi,

    I have already saw part of this article on Jeffrey Hunter blog, and it´s very good. But, I have a question, that I still could not acomplished. I want to install Oracle Clusterware with raw devices, using openfiler, like your tutorial right here, but not with OCFS2. So, Considering I´m using ASMLib( that writes some instructions on partition head, so, I don´t need to persistent names after reboot, so, don´t need to use 55- and iscsi.sh files). I used another way to create raw devices using iscsi uuid´s. But, I have already made this vm 2 times from zero and a lot of times, I had restored the snapshot to try a different approach. All the times I have the same problem when I runned the root.sh script. Have you ever used Oracle Clusterware with OCR´s and VTdisk´s over iscsi on openfiler storage?

    Thanks and Regards,

    Comment by Bruno Cantelli — August 9, 2010 @ 08:00

  2. Excellent article and thank you.

    Srinivas

    Comment by srinivas — February 27, 2011 @ 02:51


RSS feed for comments on this post.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Create a free website or blog at WordPress.com.

%d bloggers like this: