Link back: This guide is a part of the Virtual Oracle RAC project, the index to the whole project is here.
Taking care of memory
As mentioned before, now comes the time to increase the virtual machines memory to 1024MB. Take this chance to run a backup of the whole project while guests are shut down, then change Base Memory settings on both nodes. Start all the machines again.
Getting an X Server running on the PC
There are few small things to take care of before we launch Oracle Installer. Namely, we need an X Server on our PC. This can be any server of your choosing, such as Cygwin/X, Xming or other. I will be using Cygwin/X here. Make sure your X Server includes xhost utility, sometimes it is not installed by default. This is how to locate the xhost utility during instalation:
Have your X Server installed and started on your PC. If using Cygwin, run in CMD window (or via an icon):
C:\bin\cygwin2\bin\run.exe /usr/bin/bash.exe -l -c /usr/bin/startxwin.exe
This will usually start X server on display 0.0 and bring up an xterm window. If xterm window did not appear have it started manually. In this xterm window give an access control command:
$ xhost +10.10.1.11
$ xhost +10.10.1.12
This xhost command has enabled the obdn1 host to use your X Server to display x clients. I think I have just saved you a few days of frustration. The access control matters of X Server are not covered clearly in documentation and articles would suggest using X11 forwarding in SSH. The X11 forwarding in SSH will indeed take care of all the neccessary settings automaticatlly, however this setup won’t be usable by Oracle Installer. Therefore you should check that the SSH sessions you run from your PC to the RAC nodes are not using the X11 forwarding.
So, if you are using PuTTY make sure the X11 forwarding is not selected for the RAC node sessions:
As user "oracle", from the odbn1 node, from that same SSH session you have enabled SSH no-password connections (go back to refresh your memory). Ff you have restarted servers or logged in again in SSH you will need to repeat steps with ssh-agent and ssh-add as described before (on odbn1 only).
$ export DISPLAY=10.10.1.1:0
Test that your session can run some X windows application and show it on your PC’s X Server:
$ xclock &
This should spawn an xclock application and show it on your PC via X Server. If this did not work you will have to troubleshoot your setup and proceed with Oracle installation only when resolved.
Notice: The IP address we supplied in DISPLAY variable is the virtual "router" on our host PC, since X Server expect connection from 10.10.1.11, we should "talk" to it on the same network.
Having endured so far we are now going to do more exciting things.
Notice: OCFS2 package (RPM) installation instructions are in Task 7 , go back if you missed them. If you already cloned the node then do this step on both nodes.
OCFS2 will be configured to use private network i.e. hostnames as in etc/hosts:
Notice, that although we will be using the private network IP addresses we will still use hostnames as they appear on public network. I know, it is confusing. There would not be such a confusion with just one network (either private or public).
As user "root", on the odbn1 node, start a new SSH session and set the DISPLAY varaible:
# export DISPLAY=10.10.1.1:0
Now, bring up OCFS2 configuration tool to edit /etc/ocfs2/cluster.conf file:
# ocfs2console &
Select [ Cluster ] – [ Configure Nodes ]. This will start the OCFS2 Cluster Stack and bring up next dialog ("Node Configuration")
There will be a pop-up message:
Dismiss it and proceed to "Node Configuration". Add two RAC nodes as below:
When you click "Apply" the screen changes to node status and node numbers:
Exit the console and repeat the same process on the other node of the cluster supplying same exact data in the same order.
The /etc/ocfs2/cluster.conf file will look like this after configuration is done (do not just copy this file over to other node):
O2CB Cluster Service
O2CB is the OCFS2’s cluster stack of services that are listed below:
NM: Node Manager that keep track of all the nodes in the cluster.conf
HB: Heart beat service that issues up/down notifications when nodes join or leave the cluster
TCP: Handles communication between the nodes
DLM: Distributed lock manager that keeps track of all locks, its owners and status
CONFIGFS: User space driven configuration file system mounted at /config
DLMFS: User space interface to the kernel space DLM
All of the listed services are included in the single service o2cb (/etc/init.d/o2cb).
Now issue following commands to have O2CB start on boot up and have other needed attributes (on both nodes):
# /etc/init.d/o2cb offline ocfs2
# /etc/init.d/o2cb unload
# /etc/init.d/o2cb configure
Below shown non-default responses to the configuration questions (marked by arrows):
Repeat the same steps on the other node.
Issue this command to review the service status so far:
# /etc/init.d/o2cb status
Make sure that o2cb serivce will be started at proper run level:
# chkconfig –list o2cb
Levels 3 through 5 should be "on":
Format and mount the OCFS2 Filesystem
This task is to be done from one node only
# mkfs.ocfs2 -b 4K -C 32K -N 4 -L oracrsfiles /dev/iscsi/crs/part1
The following output is produced:
Now mount the newly formatted partition (that was labeled "oracrsfiles") under directory /u02:
# mount -t ocfs2 -o datavolume,nointr -L "oracrsfiles" /u02
Repeat the mount command on the other node. Verify that the file system is mounted correctly:
Repeat verification on the other node. Your output should be similar to the one below (newly mounted file system is highlighted):
Make these changes permanent by adding the following line to the /etc/fstab file on both RAC nodes:
LABEL=oracrsfiles /u02 ocfs2 _netdev,datavolume,nointr 0 0
This is how it will look like:
Adjust ownership and permissions on the new filesystem (from one node only):
# chown oracle:oinstall /u02
# chmod 775 /u02
Create directories to contain Oracle Clusterware shared files (also from one node only):
# mkdir -p /u02/oradata/racdb
# chown -R oracle:oinstall /u02/oradata
# chmod -R 775 /u02/oradata
Reboot both nodes. Check if our setup can survive the reboot. Use "mount" command to see if the new file system is still there and is mounted in the proper place. You can also use o2cb status check to make sure the serive is functioning properly (from both nodes):
# /etc/init.d/o2cb status
Configure ASMLib 2.0
Notice: ASMLib package (RPM) was added in Task 7 , go back if you missed this step. If you have already cloned the node then do this step on both nodes.
The configuration task is to be done on both nodes:
# /etc/init.d/oracleasm configure
The following picture shows your input marked by red arrows:
Create ASM disks
We are creating the ASM disks from one node only. Run following commands:
# /etc/init.d/oracleasm createdisk VOL1 /dev/iscsi/asm1/part1
# /etc/init.d/oracleasm createdisk VOL2 /dev/iscsi/asm2/part1
# /etc/init.d/oracleasm createdisk VOL3 /dev/iscsi/asm3/part1
# /etc/init.d/oracleasm createdisk VOL4 /dev/iscsi/asm4/part1
The commands above have created ASM disks in iSCSI targets that are mapped to local devices:
The newly created disks can be listed by using this command:
# /etc/init.d/oracleasm listdisks
On the other node we only need to do a scandisk to recognize the new volumes and then list them:
# /etc/init.d/oracleasm scandisks
# /etc/init.d/oracleasm listdisks
Prepare Oracle software media
Download the following from http://www.oracle.com/technetwork/database/10201linuxsoft-097986.html:
|Software Title||File Name|
Oracle Clusterware Release 2 (10.2.0.1.0)
Oracle Database 10g Release 2 (10.2.0.1.0)
Oracle Database 10g Companion CD Release 2 (10.2.0.1.0)
Now there is two ways to proceed. You can either:
copy the archives (with sftp) onto odbn1 node as user "oracle", into some directory
unzip the archives and re-pack them into an ISO image, then mount the image in VB guest
I will be using the second option as not to inflate the file system of the virtual machine (after all, it is a one time install).
So, have the archives unzipped into separate directories on your PC, then use a utility such as Infra Recorder to pack the software into a DVD ISO image, just like in the picture below:
Name this image "rac32media.iso":
Add newly created image to Virtual Media Manager and then mount it in both VB machines. Then mount it in the nodes like this:
# mount -r /dev/cdrom /media/cdrom
# cd /media/cdrom
Install cvuqdisk package
This package is needed later by Cluster Verification Utility (CVU). Issue the following commands (do it on both nodes):
# cd /media/cdrom/10201_clusterware_linux32/clusterware/rpm
# export CVUQDISK_GRP=oinstall
# rpm -iv cvuqdisk-1.0.1-1.rpm
Verify installation of the package:
# ls -l /usr/sbin/cvuqdisk
Install and run Cluster Verification Utility (CVU)
This utility will be run from odbn1 node as user "oracle". The SSH agent will have to be initiated (again) for this purpose.
Open another SSH session (we may want to use our "root" session later) and login as "oracle", then execute following commands:
$ exec /usr/bin/ssh-agent $SHELL
Install the CVU utility:
Verifying Oracle Clusterware Requirements with CVU
Once all prerequisites for running the CVU utility have been met, we can now check that all pre-installation tasks for Oracle Clusterware are completed by executing the following command as the "oracle" UNIX user account from linux1:
$ mkdir -p /home/oracle/bin/cluvfy
$ cd /home/oracle/bin/cluvfy
$ cp /media/cdrom/10201_clusterware_linux32/clusterware/cluvfy/* .
$ mkdir -p jdk14
$ unzip jrepack.zip -d jdk14
$ export CV_HOME=/home/oracle/bin/cluvfy
$ export CV_JDKHOME=/home/oracle/bin/cluvfy/jdk14
Run pre-requisites check report
$ ./runcluvfy.sh stage -pre crsinst -n odbn1,odbn2 -verbose | tee cvu_report_1.txt
Review the CVU report (cvu_report_pre.txt). There are several errors like "Check: Package existence" that can be ignored.
There is a messages in connectivity section reading "Could not find a suitable set of interfaces for VIPs". This one can be ignored too, as it is caused by a known bug documented in Metalink Note 338924.1 (has to do with non routable ranges of IP addresses).
If you get any errors other than those in my report there might be a problem with your setup.
Connectivity errors must be resolved. For reference and troublesooting see Metalink ID 338924.1 and Forum
Run the Hardware and OS Setup check report
$ ./runcluvfy.sh stage -post hwos -n odbn1,odbn2 -verbose | tee cvu_report_2.txt
Review the report, it should say "Post-check for hardware and operating system setup was successful."
The utility checks connectivity again and then checks storage, proper report will look like this:
When reports are ok and all serious problems have been resolved we can move on.
Install Oracle Clusterware
The two shared files (file groups) used by Oracle Clusterware will be stored on the Oracle Cluster File System (OFCS2) we created earlier. The two shared Oracle Clusterware file groups are:
- Oracle Cluster Registry (OCR)
File 1 : /u02/oradata/racdb/OCRFile
File 2 : /u02/oradata/racdb/OCRFile_mirror
Size : (2 * 100MB) = 200M
- CRS Voting Disk
File 1 : /u02/oradata/racdb/CSSFile
File 2 : /u02/oradata/racdb/CSSFile_mirror1
File 3 : /u02/oradata/racdb/CSSFile_mirror2
Size : (3 * 20MB) = 60MB
Issue these commands as user "oracle":
$ cd ~
$ /media/cdrom/10201_clusterware_linux32/clusterware/runInstaller -ignoreSysPrereqs &
The Oracle Installer will appear, click "Next" as in the picture below:
Have Name and Path filled as shown, click Next:
In the next screen the installer runs various checks (the checks must be all right if our manual check reports were fine) and shows the final result, click Next:
The Cluster Configuration screen appears with one node filled out already, click on Add and supply information for the other node as shown so it looks like this:
Click "Next". The Network Interface configuration screen appears (it shows subnets and network adapter names). As you can see, both interfaces are shown as "Private". We need a public interface in this configuration. Select "eth0" and click "Edit".
Designate "eth0" as public interface. Click "Ok"
Our interfaces and their purpose look like this now:
Click "Next" and OCR location dialogue appears.
Supply following OCR locations in the next screen (leave Normal Redundancy selected):
Specify OCR Location: /u02/oradata/racdb/OCRFile
Specify OCR Mirror Location: /u02/oradata/racdb/OCRFile_mirror
Supply following locations of voting disks in the next screen (again, leave Normal Redundancy selected):
Voting Disk Location: /u02/oradata/racdb/CSSFile
Additional Voting Disk 1 Location: /u02/oradata/racdb/CSSFile_mirror1
Additional Voting Disk 2 Location: /u02/oradata/racdb/CSSFile_mirror2
Summary screen appears. Click Install:
Notice how, as the install progresses, the software is being copied over to the other node in the cluster
Notice: Sometimes one of the virtual machines may crash on memory related errors (running out of usable memory on the host PC).
If this happens on odbn2 it is safe to reboot it and wait till installer continues (say 5 minutes after reboot).
If this happens on odbn1 it is more complicated. You will need to:
- cancel the installer
- restart machines if they are down
- delete subdirectories under of the /u01/app/crs and /u01/app/oracle
- shut down all VM’s
- free up host PC’s memory or reboot it
- restart VM’s
- restart the Installer
- good luck
Upon completion of the remote copy operations (ssh and tar processes) the next screen shows up:
And pop up screen shows:
Now pay attention closely.
Important: Do not close this Oracle Installer dialog until instructed to do so.
Important: run orainstRoot.sh script on both nodes, one at a time.
Important: run /u01/app/crs/root.sh script on odbn1 and wait until it completes, then proceed with the same script on the odbn2.
Output of the root.sh script on odbn1:
The /var/log/messages shows this on the first node at the completion if the script:
Now proceed with the root.sh on odbn2 node (read a note further on). The output is here:
The error at the end (libpthread.so.0) is expected, here is the manual fix to this trouble.
Edit "/u01/app/crs/bin/vipca" file on both nodes adding the following statement after line 124:
The snapshot below shows edited text with added "unset" command:
Next, edit "/u01/app/crs/bin/srvctl" file on both nodes adding same statement after line 167 so it will look like this:
To take care of this problem in future, when more software is installed into new Oracle Homes or patches applied to existsing software, the following files have to be re-edited:
Since "vipca" utility failed, we will re-run it manually. Run vipca from odbn2 (still as root):
# export DISPLAY=10.10.1.1:0
The vipca fails again with this cryptic message:
" [Error 0(Native: listNetInterfaces:)]"
The message has to do (again) with the range of IP addresses we selected for the interfaces.
Notice: If you have received another cryptic message [Exception in thread "main"] this simply means you forgot to set the DISPLAY variable.
This trouble has to be manually fixed.
Run following commands as root from odbn2 (notice that the IP addresses here are of the networks, rather than specific adapters):
# /u01/app/crs/bin/oifcfg setif -global eth0/10.10.2.0:public
# /u01/app/crs/bin/oifcfg setif -global eth1/10.10.1.0:cluster_interconnect
Let’s see now the resulting configuration of VIP network:
# /u01/app/crs/bin/oifcfg getif
So, let’s run vipca again (still on odbn2)
The following screen pops up (click on Next):
Agree wih the next screen (eth0 is indeed the public interface):
As soon as we supply first IP alias name as "odbn1-vip" the rest of fields is magically filled out. Remove the domain name from the second alias. Click "Next".
Summary screen appears. Click "Finish".
The progress screen is shown:
When this is finished and you uncross your fingers, this screen shows. Click "Ok":
The summary appears:
Exit the vipca.
Now you can back to Clusterware installer dialog and dismiss the "Execute Configuration scripts" dialogue.
The installer continues
And … fails on Cluster Verification Utility:
We ignore this message and exit the installer.
Now let’s run a few verification commands on both nodes:
# /u01/app/crs/bin/olsnodes -n
# /u01/app/crs/bin/crs_stat -t -v
# /u01/app/crs/bin/crsctl check crs
# ls -l /etc/init.d/init.*
Here are the results (one node shown, the other must show identical results):
Now we are ready to install Oracle database software.