Oracle Labs by Yuri Khazin, Oracle DBA

Virtual Oracle RAC. Task 7: Preparing Linux guest for Oracle installation – part 1


Link back: This guide is a part of the Virtual Oracle RAC project, the index to the whole project is here.

Before we run installation process for Oracle software there is some preparation needed. If not specified otherwise, the commands in this chapter are to be executed in SSH session as a user "root". If you are not familiar with UNIX and Linux, the "#" in the beginning of the command line denotes that you are logged in as user "root", the "#" is not a part of the command.

We will try to do as much work on a single node of the RAC before we clone it. After that some changes will have to be done on both nodes (or more, if you are going for more than two nodes).

Optional: Freeing up some system resources

This Linux distribution comes with many services activated by default. Some of them we are not going to use and so they can be disabled. Following commands are taking care of just that (run them as root). The change will take place after next reboot.

# chkconfig sendmail off
# chkconfig bluetooth off
# chkconfig ip6tables off
# chkconfig yum-updatesd off
# chkconfig cups off

Use this command to verify the effect of the deactivation:

# chkconfig –list

Another important change concerns default run level of the system. Under normal circumstances runlevel 3 is sufficient, while by default the system will proceed to level 5 (where it starts GUI desktop – X Windows). As we are going to use "telnet",  i.e. SSH tools to communicate with our Linux systems we do not need to run our guests at level 5. Here is how to change the default run level.

# cd /boot/grub
# vi menu.lst

Make changes in the menu.lst as shown in this picture:

Again, changes will take effect after reboot.

If you ever need to get to the X Windows Desktop again all you have to do is to issue "init 5" command as root. When you are done you can issue "init 3" command to bring the system back to level 3, no shutdown or reboot is necessary.

Special considerations for user "oracle" and UNIX groups

RAC technology requires that all nodes of the cluster have a shared access to the Oracle Clusterware software. This is achieved by using Oracle Cluster File System (OCFS2 in our case) . When using OCFS (and, if you think of this, any other shared storage solution) all nodes will authenticate themselves to the OCSF with their user IDs. In order to provide consistent access to files these IDs must match on all nodes. In other words, all users and groups that we create must have exactly same numerical IDs on different nodes. This part is easy for us, because we create our IDs only once and all our other nodes will be produced by cloning (that does not alter IDs).

Create Groups

The following O/S groups will be created:

Description Oracle Privilege Oracle Group Name UNIX Group name
Oracle Inventory and Software Owner . . oinstall
Database Administrator SYSDBA OSDBA dba
Database Operator SYSOPER OSOPER oper

# groupadd -g 501 oinstall
# groupadd -g 502 dba
#groupadd -g 503 oper

Create user "oracle"

#useradd -m -u 501 -g oinstall -G dba,oper -d /home/oracle -s /bin/bash -c "Oracle Software Owner" oracle

Verify the created user:

# id oracle

Set the password for the oracle account:

# passwd oracle

Verify That the User "nobody" exists (If you have installed the suggested Linux version you may skip this check)

1. To determine if the user exists, enter the following command:

# id nobody

2. If the user nobody does not exist (English is an interesting language if you think of that), then enter the following command to create it:

# useradd nobody

Create the Oracle Base Directory

Let’s take a look at our internal disk on odbn1 machine, there are two ways to produce the report (not all UNIX flavors support -H option):

So we have 33GB of space to work with, 2.7GB is occupied by Linux itself and its swap.

The next step is to create a new directory that will be used to store the Oracle Database software. This location is later referred to by $ORACLE_BASE environment variable.

# mkdir -p /u01/app/oracle

The "-p" option tell the mkdir command to create the whole chain of directories at once.

After the directory is created, we specify the correct owner and group (why? because we work as "root" user and newly created objects will have root’s properties):

# chown oracle:oinstall /u01/app/oracle

And now permissions:

# chmod 775 /u01/app/oracle

Create the Oracle Clusterware Home Directory

This location is later referred to by $ORA_CRS_HOME environment variable.

# mkdir -p /u01/app/crs
# chown oracle:oinstall /u01/app/crs
# chmod 775 /u01/app/crs

Create Mount Point for OCFS2 / Clusterware

# mkdir /u02
# chown oracle:oinstall /u02
# chmod 775 /u02

Create Login Script for oracle User Account

Login as user "oracle" and create the following .bash_profile:

Note: After we clone our first node into other nodes we will have to change a number of things in each of them. One of such changes is to make sure that each RAC node has a unique Oracle SID (defined in the user’s profile).

# su – oracle

Use some editor to create (or rather edit, since in OEL5 it is already there) the following profile:

.bash_profile for "oracle" User Account
# .bash_profile

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
      . ~/.bashrc
fi

export JAVA_HOME=/usr/local/java

# User specific environment and startup programs
export ORACLE_BASE=/u01/app/oracle
export ORACLE_HOME=$ORACLE_BASE/product/10.2.0/db_1
export ORA_CRS_HOME=/u01/app/crs
export ORACLE_PATH=$ORACLE_BASE/common/oracle/sql:.:$ORACLE_HOME/rdbms/admin
export CV_JDKHOME=/usr/local/java

# Each RAC node must have a unique ORACLE_SID. (i.e. racdb1, racdb2,...)
export ORACLE_SID=racdb1

export PATH=.:${JAVA_HOME}/bin:${PATH}:$HOME/bin:$ORACLE_HOME/bin
export PATH=${PATH}:/usr/bin:/bin:/usr/bin/X11:/usr/local/bin
export PATH=${PATH}:$ORACLE_BASE/common/oracle/bin
export ORACLE_TERM=xterm
export TNS_ADMIN=$ORACLE_HOME/network/admin
export ORA_NLS10=$ORACLE_HOME/nls/data
export NLS_DATE_FORMAT="DD-MON-YYYY HH24:MI:SS"
export LD_LIBRARY_PATH=$ORACLE_HOME/lib
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:$ORACLE_HOME/oracm/lib
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/lib:/usr/lib:/usr/local/lib
export CLASSPATH=$ORACLE_HOME/JRE
export CLASSPATH=${CLASSPATH}:$ORACLE_HOME/jlib
export CLASSPATH=${CLASSPATH}:$ORACLE_HOME/rdbms/jlib
export CLASSPATH=${CLASSPATH}:$ORACLE_HOME/network/jlib
export THREADS_FLAG=native
export TEMP=/tmp
export TMPDIR=/tmp

Swap Space Considerations

Exit your "oracle" user session and now you are "root" again.

We will need at least 1GB of total memory (real + swap) to install Oracle Database 10g Release 2.

Here is how to check how much we’ve got (another way is to run the "top" command):

# cat /proc/meminfo | egrep ‘MemTotal|SwapTotal’

Configure kernel parameters and shell limits for Oracle

Changes to /etc/sysctl.conf file.

You can make changes to your kernel configuration as follows (copy and paste text from "cat" to "EOF", do not include "#"):

# cat >> /etc/sysctl.conf <<EOF

kernel.shmmni = 4096

kernel.sem = 250 32000 100 128

fs.file-max = 65536

net.ipv4.ip_local_port_range = 1024 65000

EOF

Just in case I am listing the contents of /etc/sysctl.conf file after the changes:

kernel.shmmax = 4294967295

kernel.shmall = 268435456

kernel.shmmni = 4096

kernel.sem = 250 32000 100 128

fs.file-max = 65536

net.ipv4.ip_local_port_range = 1024 65000

net.core.rmem_default=1048576

net.core.rmem_max=1048576

net.core.wmem_default=262144

net.core.wmem_max=262144

Activate your changes with following command (or reboot the machine):

# sysctl -p

You can verify the new settings with this command:

#/sbin/sysctl -a

Setting Shell Limits for the oracle User

Run following command to change limits for user "oracle" (while you are still logged in as "root"):

# cat >> /etc/security/limits.conf <<EOF

oracle soft nproc 2047

oracle hard nproc 16384

oracle soft nofile 1024

oracle hard nofile 65536

EOF

# cat >> /etc/pam.d/login <<EOF

session required /lib/security/pam_limits.so

EOF

Shell settings

C shell and its variants use /etc/csh.login profile (this one is loaded before local profile). Add following lines to it:

# cat >> /etc/csh.login <<EOF

if ( \$USER == "oracle" ) then

limit maxproc 16384

limit descriptors 65536

endif

EOF

All other shells (such as Bourne, Korne and Bash) load /etc/profile, so it has to be amended like this:

# cat >> /etc/profile <<EOF

if [ \$USER = "oracle" ]; then

if [ \$SHELL = "/bin/ksh" ]; then

ulimit -p 16384

ulimit -n 65536

else

ulimit -u 16384 -n 65536

fi

umask 022

fi

EOF

Configure the hangcheck-timer Kernel Module

The module "hangcheck-timer.ko" is already installed in OEL5 and this can be verified by this command:

# find /lib/modules -name "hangcheck-timer.ko"

Set hangcheck timer parameter to values recommended by Oracle:

# echo "options hangcheck-timer hangcheck_tick=30 hangcheck_margin=180" >> /etc/modprobe.conf

To test the module and its new setting run following commands:

# modprobe hangcheck-timer

# tail -10 /var/log/messages

Set up NTP client for synchronized time on RAC nodes

Oracle installer will have a trouble if RAC nodes have different date and time showing by their clocks. The easy way to fix this problem is to set up NTP client on our machine(s). I chose to use Openfiler machine as a source of time, rather than pointing to external NTP server.

Use the "tail" command in one SSH window to verify that NTP setup is progressing properly (leave this window and come back to it later).

# tail -f /var/log/messages

In another SSH window make following changes:

# cd /etc

Make a backup of NTP settings

# cp ntp.conf ntp.conf.bak

Edit the NTP settings file to include these lines (pointing to openfiler-priv network interface):

server 10.10.1.20

restrict 10.10.1.20 mask 255.255.255.255 nomodify notrap noquery

No other NTP servers should be specified in this file, except for the default line with 127.127.1.0 address (which is optional).

Restart ntp daemon:


# service ntpd stop

# service ntpd start

If succeeded, the response will be "OK" and look like this:

The output of the "tail" command in first window may look like this by now:

We can also use "ntpq" to verify the settings (or "ntpdc" utility with "peers" command):

# ntpq -p

If you look carefully at the IP addresses you will notice that "step time server" shows IP of the "router", rather than that of the Openfiler. It is clarified when you ran "ntpq" command. Although we used Openfiler’s IP in the configuration file the command shows proper server name in the output.

As long as "st" column (Stratum) shows value other than 16 and "jitter" is reasonably small, all is working properly. The result of this setup is that odbn1 has its clock synchronized with that of the Openfiler.

RAC related changes to the system configuration files

Have this changes done as root.

Add following lines to the /etc/sysctl.conf file

net.core.rmem_default=1048576

net.core.rmem_max=1048576

net.core.wmem_default=262144

net.core.wmem_max=262144

If you are using Linux other than OEL5 U3 you may also need to add these lines:

kernel.shmmax = 4294967295

kernel.shmall = 268435456

kernel.shmmni = 4096

kernel.sem=250 32000 100 128

fs.file-max=65536

net.ipv4.ip_local_port_range=1024 65000

Have kernel load the new parameters:

# sysctl -p

Install the OCFS2

The OCFS2 software packages is supplied on CD #3 of the OEL5 U3 (but not installed by default). Have disk #3 mounted (mount the image through the Virtual Box as you did before) and issue following commands:

# mount -r /dev/cdrom /media/cdrom

# cd /media/cdrom/Server

# rpm -Uvh ocfs2-tools-1.2.7-1.el5.i386.rpm

# rpm -Uvh ocfs2-2.6.18-128.el5-1.2.9-1.el5.i686.rpm

# rpm -Uvh ocfs2console-1.2.7-1.el5.i386.rpm

Verify the installation of packages:

# rpm -qa | grep ocfs2 | sort

Install ASMLib 2.0

This library is also supplied on CD #3. Install and verify it by issuing the following commands:

# rpm -Uvh oracleasm-support-2.1.2-1.el5.i386.rpm

# rpm -Uvh oracleasm-2.6.18-128.el5-2.0.5-1.el5.i686.rpm

# rpm -qa | grep oracleasm | sort

# cd /

# eject

Install ASMLib Userspace Library

This one is not shipped along with CDs and will have to be downloaded. Get the oracleasmlib-2.0.4-1.el5.i386.rpm from Oracle.

Have this rpm copied onto the guest by means of sftp into a new directory /root/rpms

Install the rpm by running these commands:

# cd /root/rpms

# rpm -Uvh oracleasmlib-2.0.4-1.el5.i386.rpm

Looks like we are done with most of the preparations that we can do on a single node before cloning it.

Next chapter.

Advertisements

Leave a Comment »

No comments yet.

RSS feed for comments on this post.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Create a free website or blog at WordPress.com.

%d bloggers like this: