Oracle Labs by Yuri Khazin, Oracle DBA

Virtual Oracle RAC. Task 3: Openfiler – iSCSI targets setup


Link back: This guide is a part of the Virtual Oracle RAC project, the index to the whole project is here.

Now that we have Openfiler and OEL guests in Virtual Box, we can start configuring our Oracle RAC. As a general guideline I am going to use this (excellent) article – http://www.oracle.com/technology/pub/articles/hunter_rac10gr2_iscsi.html (the article gives instructions on how to build physical setup, while we are trying here to do the same in virtual environment). Roll up your sleeves.

If you have followed instructions in my other articles on installing OEL your network will differ a bit and will need some adjustments, we will cover that, of course.

The starting point is assigning names and network addresses to all our components, all hosts in our network will have exactly same /etc/hosts file, as shown below (we have placed it in our Openfiler already):

# Notice that loopback address do not have a real host name in it
127.0.0.1 localhost.localdomain localhost# Database nodes – Private Interconnect – (eth0)
10.10.1.11 odbn1-priv
10.10.1.12 odbn2-priv# Database nodes – Public Network – (eth1)
10.10.2.11 odbn1-pub
10.10.2.12 odbn2-pub

# Database nodes – Public Virtual IP (VIP) addresses – (eth1:1)
10.10.2.200 odbn1-vip
10.10.2.201 odbn2-vip

# Openfiler – private and public interfaces
10.10.1.20 openfiler-priv
10.10.2.20 openfiler

Now we do following steps:

    • Enable iSCSI services from [Services] / [Manage Services] page.
    • Verify that “iSCSI target server” service is running:

  • Network Access Configuration

Navigate to [System] / [Network Setup], notice our three network interfaces. Two of them have static IP and are active already, one needs a setup (correction – eth0 and eth1 show reversed. eth0 should be on 10.10.2.20 and eth1 on 10.10.1.20):

Click on eth2 interface and select DHCP protocol:

Confirm default MTU on next screen:

The result will look like this (actual IP not shown for eth2):

Now enable two hosts (two RAC nodes) to communicate to Openfiler as below (notice the slightly unusual netmask).

Now click on [ Clock Setup ] as shown below:

At this moment you probably ask “why don’t we use the host PC’s clock or install Guest Additions to keep guest clock synchronized with the host PC?”. Well, it is a legitimate question and here is the answer, at least what I can come up with. I have not been successful so far with Guest Additions so far, the clock has drifted wildly and some other problems were introduced (such as an occasional keyboard mapping mess up). As for synchronizing with host PC’s Windows Time service (w32time), there was not much success either. It seems that UNIX or Linux world does not understand Windows SNTP in Windows XP. So here is a sort of compromise that may work for most of the cases.

If you are setting up this project at home chances are you have access to internet and so you can use of the “pool” servers, such as 0.pool.ntp.org or similar. If you do not have access to internet but instead have a Windows Domain Server (so you are in a corporate environment) you may be lucky and our Linux guest will be able to get time from the domain controller.

Put a name or an IP of the NTP server you are going to use as a time source.  Use 10.10.1.1 to get time from domain controller (somehow Windows XP will relay this to DC). Use internet NTP server if you have access to internet.

Open an SSH window and connect to openfiler2 as root.

View /var/log/messages file, it may have these lines added after you click on “Setup synchronization”:

By this time openfiler2 will have its NTP receiving time but it will not distribute it to others. In order to allow RAC nodes to query this NTP server we have to edit /etc/ntp.conf file on openfiler2 and add two more lines to it:

restrict 10.10.1.11 nomodify notrap

restrict 10.10.1.12 nomodify notrap

After that you will need to restart the NTP service:

# service ntpd restart

If you have trouble with this NTP setup you should examine /var/log/messages and /etc/ntp.conf files. Use this source of information on NTP servers and clients to fix your problem. Notice, that RAC nodes will be able to sync their clocks from openfiler2 even if openfiler2 itself won’t be able to get precise time from other sources (as long as it permits to query itself), so this resolves a potential problem with date/time and Oracle installer later on.

  • Physical Storage

Navigate to [Volumes]. Most probably you will see a picture like this:

Navigate to [Block Devices] section (see below):

Click on the link for “hdb”, scroll to “Create a partition in /dev/hdb” and click on “Create” thus accepting defaults:

The result of this action presented below:

Now navigate to [Volumes] / [Volume Groups] and create a new volume group as below:

Not tired yet? Navigate to [Volumes] / [Add Volume] and scroll down to “Create a volume” in “rac1” and use the table below (borrowed from original article mentioned in the beginning, numbers are slightly changed though).

iSCSI / Logical Volumes
Volume Name Volume Description Required Space (MB) Filesystem Type
racdb-crs racdb – Oracle Clusterware 2,048 iSCSI
racdb-asm1 racdb – ASM Volume 1 8,092 iSCSI
racdb-asm2 racdb – ASM Volume 2 8,092 iSCSI
racdb-asm3 racdb – ASM Volume 3 8,092 iSCSI
racdb-asm4 racdb – ASM Volume 4 8,092 iSCSI

Here is the picture of first volume being created (after this one you will have to click on [Add Volume] again):

In the end you will get a setup like this:

I am getting tired a bit but hold on a little longer, we are getting there. It’s actually getting quite exciting as we are going to create iSCSI targets now.
Navigate to [Volumes] / [iSCSI Targets] and proceed to add five iSCSI targets with following names (system will suggest default names that you will change, it is only a matter of aesthetics). The names will be as below:

We are not going to change any settings for our targets (leave all defaults).
Select first target and click “Change” (this is going to affect sub-tabs on this page). Click on “LUN Mapping” sub-tab. You will see this (look at it for a while and it will make sense):

If you did not guess yet, you will have to click on “Map” button thus linking LUN “/dev/rac1/racdb-crs” with iSCSI target “iqn.2006-01.com.openfiler:racdb.crs”.
After that proceed to “Network ACL” sub-tab and grant our RAC nodes the access to this target as below:

The last sub-tab – “CHAP”  is not relevant.

Now go to first sub-tab and select next target, then link it with appropriate LUN and grant access. Repeat this process until you linked them all.  At this point we are done with Openfiler.

I am sending special thanks to Jeffrey Hunter, whose article was guiding me through this setup so far.

Next chapter.

Advertisements

3 Comments »

  1. Hi

    First of all would like to thank and congratulate you for the great job done and the efforts taken for putting all this info.

    Its great help … Thanks

    Diogo

    Comment by Diogo Fernandes — August 15, 2010 @ 16:13

    • I have posted a “correction – eth0 and eth1 show reversed. eth0 should be on 10.10.2.20 and eth1 on 10.10.1.20”

      Sorry for confusion. Later on I will replace the image. The hosts file is correct though.

      Comment by oraclelabs — February 15, 2011 @ 14:15

  2. openfiler (2.3 and 2.99) seems to have an issue with dashes in volume-names. Created them as racdb1 racdb2 to actually be able to use them.

    Comment by jj — February 18, 2013 @ 18:40


RSS feed for comments on this post.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Blog at WordPress.com.

%d bloggers like this: