Overview:

Oracle RMAN uses the Serial Backup Tape (SBT) API to perform backup to tape devises.  The NetBackup Oracle extension is an implementation of the SBT API.

Upgrading the SBT API, can present some challenges for an application that runs 24 x 7.  The information below should be reviewed and well understood before planning the installation or upgrade of NetBackup on an Oracle host.

The nature of running processes is that, by default, external references are resolved and the relevant shared object libraries read from disk and mapped into the running process space only once during the life of a process.  Thus a process that runs continuously and performs a backup every day typically does not reload libraries before each backup.  Consequently, the only way to force the process to load an updated copy of a library is by stopping and restarting the process.  Hence the challenge to a 24 x 7 application.

Recommendations:

Follow these steps to perform a successful upgrade of NetBackup on an Oracle client host.  This applies to upgrading the NetBackup Oracle extension and the NetBackup Client whose libraries are used by the extension.  Prior to NetBackup 7.0, these are separately installed components and both should always be upgraded at the same time and to the same maintenance pack or release update level.  Starting with NetBackup 7.0, the NetBackup Client install automatically includes the NetBackup for Oracle extension.

Please note that all references to ‘sbt operations’ encompasses backup, restore, and catalog maintenance operations.

1) Stop all processes for the Oracle instances on the host.  Some may have the old libraries mapped into process space.  If there is more than one instance and all are using NetBackup, then all should be stopped.

2) Stop the Oracle listener process if sbt operations have been performed using TNS aliases since NetBackup was last installed or upgraded.  In that configuration, the Oracle listener spawns the process that will do the sbt operation and it too will likely have the old libraries mapped into process space.

3) On HP-UX, the files on disk are the backing store for the running process and may be locked, causing any attempt to overwrite the files to fail.  Check if the files are in use and terminate any processes that are using them prior to updating the libraries.

$ fuser /usr/openv/lib/libxbsa*
$ fuser /usr/openv/netbackup/bin/libobk*

4) On AIX, the old library may already be in the library cache.  New or existing processes will look in the library cache first and may not load the new libraries from disk when resolving external references.  If all the Oracle processes noted above have been halted, clear the cache.

$ /usr/sbin/slibclean

5) On Windows, locate all ‘*xbsa*.dll‘ and ‘orasbt.dll‘ files and delete them.  The install will reinstall the new copies in the appropriate places and the older ones will no longer be inadvertently found higher in the search PATH when resolving external references.

6) Perform the install or upgrade per the NetBackup software distribution instructions.

7) After the install, inspect the output from the following commands to confirm that the expected version of the files are installed.

$ cd /usr/openv
$ cat netbackup/bin/version
$ cat share/*oebu*
$ ls -1 lib/libxbsa* netbackup/bin/libobk* \
 | while read fn ; do
   netbackup/bin/goodies/support/versioninfo -f $fn
 done

Note that the versioninfo program has been included in the NetBackup server distribution since NetBackup 6.0, but was not added to the client distribution until the 6.5.4 release update.  It can be copied from a server of the same platform type as the client.

On Windows, locate the files and check their properties.

8) Following the install, ensure that Oracle is properly using the newly installed libraries by follow the steps in TECH72307 in the Related Articles section.

Final Notes:

Newer versions of Oracle (9i and above) should dynamically load and unload the SBT library as needed, but in rare instance reportedly do not.  The following recommendations have been found to be useful in the past.

A) Consistently use SBT_LIBRARY for all SBT operations.  This will cause an explicit dlopen system call to locate and read the library file when the channel is allocated.  Then when the channel is released, an explicit dlclose system call will unload the library from the process space so that it can be reloaded from disk, when the channel is allocated for the next backup or restore.  I.e.

ALLOCATE CHANNEL … TYPE SBT_TAPE PARMS=’SBT_LIBRARY=/usr/openv/netbackup/bin/<appropriate_libobk>’;

On AIX, be aware that the old library will still be referenced by the library cache.  But if all sbt operations specified SBT_LIBRARY and are complete, the use count will be 0 so slibclean will remove it from the cache.

B) Avoid using a TNS alias to connect to the target database when the database is local to the host that is running RMAN.  Using an alias causes the Oracle listener to create the Oracle server process.  The listener may be running as a different user than the instance to backup or restore, which may have a different $ORACLE_HOME, which will cause a different path to be searched for libobk, which may cause an unexpected libobk to be loaded and used.  See the Related Articles for details regarding Oracle 11g.

• TECH72307   How to confirm that Oracle is loading the correct NBU Oracle extension library files for use?

• TECH194511   SBT library failed to load when using the remote connections to backup Oracle 11gR2 RAC on AIX 6.1

When the Oracle RMAN program performs a backup, restore, or catalog maintenance operation using the SBT API, it will utilize a libobk* shared object library or orasbt.dll.  This library is provided by third-party backup software, including the NetBackup (NBU) Oracle extension.  The NBU Oracle extension is dependent upon the xbsa library provided with the NBU Client.

Below is a process for confirming if the correct library files are installed and being utilized.  These examples are for Unix, but the process is the same for Windows and the details are at the bottom of this document.

1) Shutdown the Oracle instance.

2) If TNS aliases have or will be used by RMAN to connect to the instance(s) then also shutdown the listener.

3) Confirm all Oracle processes are down.

$ ps -ef | grep -i ora

4) On AIX, also clear the library cache.  This will only work if all Oracle process that utilize the libobk are down and the library use counter has decremented to 0.

$ /usr/sbin/slibclean

Note: Confirming that the Oracle process are down is significant!  Once the Oracle instance or listener starts and attempts an SBT API operation, it loads the then current library files from disk into memory.  The running process should unload the library when not needed and then load a new copy when needed, but in rare instances may not.  If that happens, the Oracle instance may remain ignorant of updated copies on disk and continue to use the older copy already loaded into process space.  See TECH72419 in the Related Articles section for additional details.

5) Capture the last access times on the library files defined by Oracle, NBU Oracle, and NBU Client.

(
ls -lu $ORACLE_HOME/lib*/libobk*
ls -lu /usr/openv/netbackup/bin/libobk*
ls -lu /usr/openv/lib/libxbsa*
) > /tmp/nbu-lib-access.before.out

6) Restart the Oracle instance.

7) Perform a backup, restore or catalog maintenance operation using RMAN.

8) Capture the last access times on the library files again.

(
ls -lu $ORACLE_HOME/lib*/libobk*
ls -lu /usr/openv/netbackup/bin/libobk*
ls -lu /usr/openv/lib/libxbsa*
) > /tmp/nbu-lib-access.after.out

9) Compare the output files from steps 5 and 8.

The access time on one of the NBU libobk.* files and one of the NBU libxbsa files should have been updated.  The access time on the libobk.* in one of the lib, lib32, or lib64 subdirectories below $ORACLE_HOME may also have been updated.

If the access times did not update on the expected library files then either the Oracle instance configuration or the RMAN PARMS statement in the backup/restore/maintenance script specifies an alternate location for SBT_LIBRARY or another libobk* file exists higher on the LD_LIBRARY_PATH (Solaris & Linux), SHLIB_PATH (HP-UX), or LIBPATH (AIX).  The DBA will be familiar with the Oracle library load search process and can make the adjustments so that it uses the correct file.

If the access times updated on an unexpected files, then the DBA will also need to correct the Oracle library load search process.  This may involve specifying SBT_LIBRARY, deleting older libraries that are higher on the search path, or symbolically linking the instance to the NBU appropriate libobk.  To build the correct symbolic links, use this script.

$ /usr/openv/netbackup/bin/oracle_link

If the access times did not update on any of the libobk files in the Oracle or NBU directories updated, then it is likely that RMAN is connecting to an Oracle instance running on another host instead of on this host.  The DBA should check if the target instance is being accessed via a TNS alias and if the alias is resolving correctly.

Note: Starting with NetBackup 6.5.6 (or any hotfix to support Oracle 11g R2), the NBU libobk is statically linked with the xbsa library and the update time on the xbsa library will not change during the operations above.

10) After the symbolic links have been corrected, repeat steps 1-9 to ensure the correct library file is in use.

11) If the correct files are being accessed, but the libraries still will not load, then check the NetBackup Database Compatibility matrix to ensure that the platform, architecture (32 or 64 bit), and version of Oracle that is in use is also supported by the version of NetBackup that is installed.

12) For the next few days or weeks, enable the dbclient debug log and monitor for these lines.  The build dates for the xbsa library and NetBackup for Oracle should be the same or only a few days different.  If they differ significantly, investigate if a mismatched older library is present and in use by some process.

$ cd /usr/openv/netbackup/logs/dbclient
$ egrep ‘NetBackup XBSA Interface|NetBackup for Oracle’ log.??????
08:30:33.508 [11019] <4> VxBSAInit: Veritas NetBackup XBSA Interface – 7.1  2011020313
Veritas NetBackup for Oracle – Release 7.1 (2011020313)
08:34:13.996 [12828] <4> VxBSAInit: Veritas NetBackup XBSA Interface – 7.1  2011020313
Veritas NetBackup for Oracle – Release 7.1 (2011020313)
09:04:48.405 [26337] <4> VxBSAInit: Veritas NetBackup XBSA Interface – 7.1  2011020313
Veritas NetBackup for Oracle – Release 7.1 (2011020313)

For comparison, these are the build dates for libxbsa and libobk for the NetBackup 6.5 and 7.x release.

Veritas NetBackup XBSA Interface – 6.5  2007111605 6.5.1
Veritas NetBackup for Oracle – Release 6.5 (2007111606)

    Veritas NetBackup XBSA Interface – 6.5  2008052300 6.5.2
Veritas NetBackup for Oracle – Release 6.5 (2008052301)

    Veritas NetBackup XBSA Interface – 6.5  2009120409 6.5.3
(No NB Oracle fixes in 6.5.3, should be using NB_ORA_6.5.2.)

    Veritas NetBackup XBSA Interface – 6.5  2009050105  6.5.4
Veritas NetBackup for Oracle – Release 6.5 (2009050106)

    Veritas NetBackup XBSA Interface – 6.5  2009110613  6.5.5
(No NB Oracle fixes in 6.5.5, should be using NB_ORA_6.5.4.)

    Veritas NetBackup XBSA Interface – 6.5  2010042404 6.5.6
Veritas NetBackup for Oracle – Release 6.5 (2010042405)

    Veritas NetBackup XBSA Interface – 7.0  2010010418 7.0
Veritas NetBackup for Oracle – Release 7.0 (2010010418)

    Veritas NetBackup XBSA Interface – 7.0  2010070723 7.0.1
Veritas NetBackup for Oracle – Release 7.0 (2010070723)

    Veritas NetBackup XBSA Interface – 7.1  2011020313 7.1
Veritas NetBackup for Oracle – Release 7.1 (2011020313)

    Veritas NetBackup XBSA Interface – 7.1  2011061213 7.1.0.1
Veritas NetBackup for Oracle – Release 7.1 (2011061213)

    Veritas NetBackup XBSA Interface – 7.1  2011082510 7.1.0.2
Veritas NetBackup for Oracle – Release 7.1 (2011082510)

If the build dates are still not correct, then enable the RMAN Channel Trace and inspect the resulting trace file in the user dump destination to see where the library is being loaded from.  E.g.

$ cat udump/oracle9_ora_21811.trc
…snip…
try loading : libobk.so
Loaded (/app/oracle/lib/libobk.so)

Windows Specific Information: