Thursday, September 29, 2011

Starting/Stopping a RAC Instance

2-node RAC cluster setup with instances RAC1 on node rac01 and RAC2 on node rac02 and ORACLE_UNQNAME=RAC.  Need to shutdown RAC2 on node rac02 so the server can brought down for system maintenance.

Stop the instance on rac02 first as oracle
$ cd $ORACLE_HOME/bin

$ srvctl stop instance -d RAC -i RAC2

Check the status of the instance
$ srvctl status database -d RAC
Instance RAC1 is running on node rac01
Instance RAC2 is running on node rac02

Now stop the cluster services on rac02 as root

# cd /opt/app/11.2.0/grid/bin

# ./crsctl stop cluster -n rac02

After maintenance cycle is complete, start the cluster on node rac02
# ./crsctl start cluster -n rac02

Start the database instance RAC2 on node rac02
$ cd $ORACLE_HOME/bin
srvctl start instance -d rac -i RAC2

Start the EM console is necessary
$ emctl start dbconsole

Log location for OEM are located here:
$ORACLE_HOME/rac01_RAC/sysman/log
and
$ORACLE_HOME/rac02_RAC/sysman/log

Friday, September 23, 2011

Network Based Oracle Enterprise Linux Install

Using the HHTP method, create the following directories on 192.168.2.103

# mkdir /var/www/html/network-install
# mkdir /var/www/html/network-install/RPM
# mkdir /var/www/html/network-install/ISO

Copy the files from my OEL 5u4 DVD

# cp -r /media/Enterprise\ Linux\ dvd\ 20090908/* /var/www/html/network-install/RPM

Start the Apache web server
# httpd -k start

Start Firefox or a web browser and point to:
http://192.168.2.103/network-install/RPM/

Files should be visible

Next, create a Kickstart boot CD.  From disk 1, I created an image of boot.iso located in the images directory.  I used a Windows box since I had the disk 1 ISO on my local Windows machine.  Use this to boot the new server.

Boot system, then at the boot prompt type
boot: linux askmethod

Installation Method select HTTP

Networking Device, choose eth0

Configure TCP/IP, in Enable IPv4 support, select Manual Configuration and disable IPv6 support

IPv4 address: 192.168.2.102/255.255.255.0
Gateway: 192.168.2.1
Leave Nameserver blank

HTTP Setup

Web site name: 192.168.2.103
Enterise Linux directory: /network-install/RPM

Proceed with the install as usual.

REFERENCES:

http://www.linuxhomenetworking.com/wiki/index.php/Quick_HOWTO_:_Ch25_:_Network-Based_Linux_Installation

http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/3/html/System_Administration_Guide/s1-kickstart2-startinginstall.html

Thursday, September 22, 2011

Kill a Port

Trying to start an application, but the port it needs to bind is open.  Usually this happens when the application crashes and fails to close any of its open ports.

Assume that I'm looking to 1780, since that what's my application binds to.  Use lsof to list any open files used by that port

# lsof  | grep 1780

Who's using that port?

# fuser -v 1780/tcp

                     USER        PID ACCESS COMMAND
1780/tcp:            root      22516 F.... uxwdog
                          redhat    22518 F.... ns-httpd

Got the process ID, so kill the process

# fuser -vk 1780/tcp

Check to make sure it's gone...

# fuser -v 1781/tcp

Creating a Bootable USB Stick

I've downloaded OEL 5u4 and created an ISO image on a DVD drive.  I've installed this on of my test servers but come to find out that the other server does not have a DVD drive but a standard CD-ROM.  Goal is to try and create a bootable USB stick.

Put DVD into drive.  If DVD mount, un-mount it.  OEL 5u4 automatically mount the DVD.

# umount /dev/hda

Put USB stick in USB and copy the contents of the DVD to the USB stick

# dd if=/dev/hda of=/media/disk/install.iso

This should work...

Friday, September 9, 2011

Creating a CentOS Repository for a 2-Node RAC

Current setup is a 2-node 11g Oracle RAC using a Dell desktop as a  storage server.  The two RAC nodes (rac01, rac02) access the storage server (nas01) using NFS.  I used instructions from the following to build my RAC setup:

http://www.oracle-base.com/articles/11g/OracleDB11gR2RACInstallationOnLinuxUsingNFS.php

I'm going to build my private CentOS mirror on nas01.  I used instructions from the following to build my private CentOS mirror:

http://nerdicism.com/2009/04/how-to-setup-up-a-private-centos-mirror/

I'm then going to update all three servers from CentOS 5.3 to CentOS 5.6.

Before I begin, I'm going to shutdown the database and cluster services on rac01 and rac02.  I'll do this from rac01.


# su - oracle
$ srvctl stop database -d RAC

Check the status
$ srvctl status database -d RAC
Instance rac01 is not running on node rac01
Instance rac02 is not running on node rac02

Stop the listener and scan_listener
$ srvctl stop listener
$ srvctl stop scan_listener

Check the status of the listener and scan_listener
$ srvctl status listener
Listener LISTENER is enabled
Listener LISTENER is not running

$ srvctl status scan_listener
SCAN Listener LISTENER_SCAN1 is enabled
SCAN listener LISTENER_SCAN1 is not running

$ exit

Shutdown cluster ready services (CRS)
# cd /opt/app/11.2.0/grid/bin
./crsctl stop crs
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac01'
CRS-2673: Attempting to stop 'ora.crsd' on 'rac01'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rac01'
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'rac01'
CRS-2673: Attempting to stop 'ora.rac01.vip' on 'rac01'
CRS-2677: Stop of 'ora.rac01.vip' on 'rac01' succeeded
CRS-2673: Attempting to stop 'ora.rac02.vip' on 'rac01'
CRS-2677: Stop of 'ora.scan1.vip' on 'rac01' succeeded
CRS-2677: Stop of 'ora.rac02.vip' on 'rac01' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'rac01'
CRS-2673: Attempting to stop 'ora.eons' on 'rac01'
CRS-2677: Stop of 'ora.ons' on 'rac01' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'rac01'
CRS-2677: Stop of 'ora.net1.network' on 'rac01' succeeded
CRS-2677: Stop of 'ora.eons' on 'rac01' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rac01' has completed
CRS-2677: Stop of 'ora.crsd' on 'rac01' succeeded
CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'rac01'
CRS-2673: Attempting to stop 'ora.ctssd' on 'rac01'
CRS-2673: Attempting to stop 'ora.evmd' on 'rac01'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rac01'
CRS-2677: Stop of 'ora.cssdmonitor' on 'rac01' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rac01' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'rac01' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'rac01' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rac01'
CRS-2677: Stop of 'ora.cssd' on 'rac01' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rac01'
CRS-2673: Attempting to stop 'ora.diskmon' on 'rac01'
CRS-2677: Stop of 'ora.gpnpd' on 'rac01' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rac01'
CRS-2677: Stop of 'ora.gipcd' on 'rac01' succeeded
CRS-2677: Stop of 'ora.diskmon' on 'rac01' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rac01' has completed
CRS-4133: Oracle High Availability Services has been stopped.

Note that only rac01 is listed because at the time I shutdown the services, node rac02 was down.

Before beginning, I need to confirm the current CentOS version.
# cat /etc/redhat-release
CentOS release 5.3 (Final)

# lsb_release -a
LSB Version:    :core-3.1-amd64:core-3.1-ia32:core-3.1-noarch:graphics-3.1-amd64:graphics-3.1-ia32:graphics-3.1-noarch
Distributor ID: CentOS
Description:    CentOS release 5.3 (Final)
Release:        5.3
Codename:       Final

On nas01, create the directory to hold all the RPMs
# mkdir -p /var/www/html/centos/5/os/x86_64
# mkdir -p /var/www/html/centos/5/updates/x86_64

Start Apache for my private mirror on nas01.
# httpd -k start


Download CentOS 5.6 OS from a mirror site and copy over to the /var/www/html/centos/5/os/x86_64 directory on nas01.  In this case, there are 8 iso files.  Download all the updates and copy over to /var/www/html/centos/5/updates/x86_64 on nas01.

The updates and OS ISOs were downloaded from the following locations:

http://www.mirror.facebook.net/centos/5.6/os/x86_64/RPMS/

http://www.mirror.facebook.net/centos/5.6/updates/x86_64/RPMS/

I used wget to download the RPMs from the updates directory as follows:

# wget -r -l1 --no-parent -A.RPM http://www.mirror.facebook.net/centos/5.6/updates/x86_64/RPMS/

From the wget documentation:

"More verbose, but the effect is the same. ‘-r -l1’ means to retrieve recursively (see
Chapter 3 [Recursive Download], page 24), with maximum depth of 1. ‘--no-parent’
means that references to the parent directory are ignored (see Section 4.3 [Directory-Based
Limits], page 27), and ‘-A.gif’ means to download only the gif files. ‘-A "*.gif"’ would
have worked too."

Extract the files from ISOs as follows:

# cd /var/www/html/centos/5/os/x86_64
# mkdir /tmp/mnt
# mount -o loop CentOS-5.6-x86_64-bin-1of8.iso /tmp/mnt
# cd /tmp/mnt
# cp -rf * /var/www/html/centos/5/os/x86_64/
# cd -
# umount /tmp/mnt
# mount -o loop CentOS-5.6-x86_64-bin-2of8.iso /tmp/mnt
# cp -rf /tmp/mnt/* .
# umount /tmp/mnt
# mount -o loop CentOS-5.6-x86_64-bin-3of8.iso /tmp/mnt
# cp -rf /tmp/mnt/*
# umount /tmp/mnt
# mount -o loop CentOS-5.6-x86_64-bin-4of8.iso /tmp/mnt
# cp -rf /tmp/mnt/* .
# umount /tmp/mnt
# mount -o loop CentOS-5.6-x86_64-bin-5of8.iso /tmp/mnt
# umount /tmp/mnt
# mount -o loop CentOS-5.6-x86_64-bin-6of8.iso /tmp/mnt
# cp -rf /tmp/mnt .
# umount /tmp/mnt
# mount -o loop CentOS-5.6-x86_64-bin-7of8.iso /tmp/mnt
# cp -rf /tmp/mnt .
# umount /tmp/mnt
# mount -o loop CentOS-5.6-x86_64-bin-8of8.iso /tmp/mnt
# umount /tmp/mnt

Modify the /etc/yum.repos.d/CentOS-Base.repo file

# cd /etc/yum.repos.d
# vi CentOS-Base.repo

[base]
name=CentOS-$releasever - Base
# Add this line...
baseurl=http://192.168.2.103/centos/$releasever/os/$basearch/
# Comment out this line...
#mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=os
#baseurl=http://mirror.centos.org/centos/$releasever/os/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-5

#released updates
[updates]
name=CentOS-$releasever - Updates
# Add this line...
baseurl=http://192.168.2.103/centos/$releasever/updates/$basearch/
# Comment out this line...
#mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=updates
#baseurl=http://mirror.centos.org/centos/$releasever/updates/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-5

Make sure that files are available.  Open a web browser using:
http://192.168.2.103/centos/5/os/x86_64/
http://192.168.2.103/centos/5/updates/x86_64/

Where 192.168.2.103 is the address of my private mirror.

Run yum clean and update
# yum clean all
Loaded plugins: fastestmirror
Cleaning up Everything
Cleaning up list of fastest mirrors

# yum update

Running yum update on nas01 caused the update to hang.  I then killed the process by hitting Ctrl+c.  Then I tried rerunning yum update but got the following error:

"There are unfinished transactions remaining.  You might consider running yum-complete-transaction to finish them."

yum-complete-transaction was not installed on nas01.  I installed it as follows:

# yum install yum-utils-1.1.16-14.el5.centos.1.noarch.rpm

which was installed from my private mirror.

I ran yum-complete-transaction which completed the one remaining transaction I had.

I re-ran yum update and it completed successfully.