How to enable logging to troubleshoot NetBackup

Basic troubleshooting for Veritas Netbackup

Below are some basic areas which any newbie use to diagnose the problem.

1.) Space

Check the space in the Disk Storage Units, Tapes, Disk arrays etc


2.) Network Connectivity

Check the physical network connectivity via ping command


3.) Services

Verify that the Netbackup Services is being run at Netbackup Server from:

Netbackup Administration Console-->Activity Monitor-->Services

Verify that the Netbackup Client Service is started at client from:

Start-->Settings-->Control Panel-->Administrative Tools-->Services


4.) Check the Name Resolution


On the Master Server:


• …Program Files\Veritas\NetBackup\bin>bpclntcmd.exe -hn Client name

On the client:

• ….Program Files\Veritas\NetBackup\bin>bpclntcmd.exe –pn

• ….Program Files\Veritas\NetBackup\bin>bpclntcmd -ip <clients ip address>


5.) Create Log folders

After creating the below log folders the logs start creating which we can help us to troubleshoot the problem.

• Create on client bpbkar folder under:

<install path>\netbackup\logs

bpbkar is creates a list of files which backups.

• Create bpbrm,bptm,bpcd folders under ….Program Files\Veritas\NetBackup\logs

• Go to Netbackup Administration Console-->Host Properties-->Master Server-->Properties-->Logging-->under Process Specific override--> select 5 from the drop down box for BPBRM,BPCD

• At client side create the folders bpcd and dbclient under
<install path>\netbackup\logs

• Go to Backup, Archive and restore window-->File-->Netbackup Client Properties-->Troubleshooting-->write 5 in the verbose and 9 in the database box

Setting up logs in NetBackup


For a given issue, it may be necessary to gather multiple logs.  This MUST cover the time the issue happens.
If an additional log is required, that has to be created, then ALL the logs must be supplied again.

There are two types of logs in NetBackup.  Legacy logs and VX logs.


1) Creating Legacy Logs
2) Setting Verbose Level for Legacy Logs
3) Collecting Legacy Logs
4) Creating VX Logs and setting the log level
5) Collecting VX logs
6) Volmgr Logs


1) Creating Legacy Logs
************************

These are created in either

Unix
/usr/openv/netbackup/logs/
/usr/openv/volmgr/debug/

Windows
<install path>\veritas\netbackup\logs
<install path>\veritas\volmgr\debug


For example to create bptm log, simply create a directory called the <process> name.

mkdir /usr/openv/netbackup/logs/bptm

A newly created log will not log anything /detect a change of verbose level until the process is restarted.  For logs such as bptm, this will be when the next backup runs.  Other logs such as bprm and bpdm may require a restart of the NBU services.  I say 'may', if the process starts a child process, then this would write to a newly created log or pick up a verbose level change.


2) Setting Verbose Level for Legacy Logs
****************************************

There are two ways this can be done :

(Unix) 

To increase the verbose level of all logs (except vault)
Add the entry VERBOSE = <level> into /usr/openv/netbackup/bp.conf.  <level> is a value between 0 and 5, with 5 being the highest.


(Windows)

On the server you are gathering the logs from, run the BAR GUI
From the File menu, select Client Properties and in the pop-up window, goto the Troubleshooting tab.
Set General to 2 and Verbose to 5


3) Collecting Legacy Logs
*************************

The log file is simply found in the <process> name directory.  There is one log per day.

The name of the log file will be log.<date>

If you are sending multiple log files in, they will all have the same name.  Please therefore rename the log files to :

<process>.log.<date>


4) Creating VX Logs and setting the log level
**********************************************

These are more complex, and have to be set with specific commands.  NOTE: Some of these logs, for example, 'mds' do NOT create a log file.  Instead the lines are entered in to other log files.  In the case of mds (143), it logs into EMM (111).


The vxlogs cover various processes, for example, nbemm, nbrb, nbjm, nbrb, mds


To set these up on either Unix or Windows, use this command :

vxlogcfg -a -p 51216 -o <oid> -s DebugLevel=<1-6> -s DiagnosticLevel=<1-6>

For example, to set the EMM and MDS logs to levels 6 and 6 use

vxlogcfg -a -p 51216 -o 111 -s DebugLevel=6 -s DiagnosticLevel=6
vxlogcfg -a -p 51216 -o 143 -s DebugLevel=6 -s DiagnosticLevel=6


To confirm the log level has been set, simply look in the nblog.conf file, which is located in the netbackup diorectory.

(NOTE: DO NOT EDIT THIS FILE MANUALLY)


5) Collecting VX Logs
*********************

To collect the vx logs, use the nbcplogs command.  This copies the raw logs, which is the preference of Technical Support.

NOTE:  The destination directory MUST be empty.


nbcplogs --no-nbsu -d 2hrs --logs nbemm,nbjm /tmp/logs   (Ex.  Coleect the past 2 hrs of logs, RELATIVE, to when the command is run )
 
nbcplogs --no-nbsu -s 07/11/2012-10:17:58 -e 07/11/2012-12:17:58 --logs nbjm,nbpem /tmp/logs (Collect the logs between two times -s <start> -e <end>  )

In these examples, the nbpem and nbjm logs would be copied to /tmp/logs


For details of using vxlogview please see TN:  http://www.symantec.com/docs/TECH75805


6) Volmgr Debug Logs
********************

These are very similar to legacy logs, the difference being the location and the verbose setting.
There is no value for verbose level, simply it is turned up by adding the work VERBOSE to a line in the vm.conf file.

To turn on Media Manager logging:


UNIX:

Add VERBOSE to the /usr/openv/volmgr/vm.conf file.  If this file does not exist, just create it.
If necessary, create the directory /usr/openv/volmgr/debug

mkdir /usr/openv/volmgr/debug/acssi
mkdir /usr/openv/volmgr/debug/acsd
mkdir /usr/openv/volmgr/debug/robots
mkdir /usr/openv/volmgr/debug/daemon
mkdir /usr/openv/volmgr/debug/ltid
mkdir /usr/openv/volmgr/debug/oprd
mkdir /usr/openv/volmgr/debug/reqlib
mkdir /usr/openv/volmgr/debug/tpcommand

The following empty files increase the details logged further

touch /usr/openv/volmgr/DRIVE_DEBUG
touch /usr/openv/volmgr/ROBOT_DEBUG
touch /usr/openv/volmgr/AVRD_DEBUG
touch /usr/openv/volmgr/SSO_DEBUG


Restart ltid :

/usr/openv/volmgr/bin/stopltid
/usr/openv/volmgr/bin/ltid -v


Windows:

Add VERBOSE to the <install path>\veritas\volmgr\vm.conf file.  If this file does not exist, just create it.

If necessary, create the directory /usr/openv/volmgr/debug

<install path>\veritas\volmgr\debug/acsssi
<install path>\veritas\volmgr\debug/acsd
<install path>\veritas\volmgr\debug/robots
<install path>\veritas\volmgr\debug/daemon
<install path>\veritas\volmgr\debug/ltid
<install path>\veritas\volmgr\debug/oprd
<install path>\veritas\volmgr\reqlib
<install path>\veritas\volmgr\debug\tpcommand

Create the following empty files to increase the details logged further
Ensure that windows does not craete a 'suffix', for example .txt

<install path>\veritas\volmgr\DRIVE_DEBUG
<install path>\veritas\volmgr\ROBOT_DEBUG
<install path>\veritas\volmgr\AVRD_DEBUG
<install path>\veritas\volmgr\SSO_DEBUG


Restart ltid :

<install path>\veritas\volmgr\bin/stopltid
<install path>\veritas\volmgr\bin/ltid -v


NOTE:

The 'OIDs' for the Unified /VX logs can be determined by looking in the nblog.conf file in the /usr/openv/netbackup directory.  Be careful not to edit the file.

The default logs levels for the Unified /VX log levels can be found in the nblog.conf.  These are set as follows:

NetBackup Server : Default.DiagnosticLevel=6 / Default.DebugLevel=1
NetBackup Appliance : Default.DiagnosticLevel=6 / Default.DebugLevel=5



For ORACLE and RMAN  Backup Issue :-

How to enable logging to troubleshoot NetBackup for Oracle RMAN backup and Restore

Solution

The majority of cases at Veritas require the collection of log files in order to determine the nature of the backup or restore problem. Given the comprehensive nature of NetBackup (tm), knowing which logs to collect and which logs are unnecessary can present a daunting task.

This TechNote is designed to assist a user in determining what logs a Support Engineer needs to see in order to expedite the troubleshooting of an Oracle case. While there are cases where logs other than the ones documented are required for resolution, this TechNote should cover the requirements for nearly all cases.

By default, log files are placed in the <install_path>\veritas\netbackup\logs (Windows) and /usr/openv/netbackup/logs (UNIX/Linux) directories on a NetBackup server.


To enable logging on any of the NetBackup processes, a subdirectory must be created in the NetBackup logs directory on the NetBackup host.


Unix: /usr/openv/netbackup/logs

Windows: <install_path>\Veritas\NetBackup\logs


The following log folders should be created on the Oracle client server:

dbclient: 

This log contains debugging information and execution status for the NetBackup Oracle client processes.  Ensure that the Oracle user has permissions to write to this directory.

bpdbsbora: 

This log contains debugging information and execution status for the bpdbsbora utility which saves, retrieves and executes templates.

bphdb: 

This log contains debugging information for the process that executes the backup/restore script or template.  The obk_stdout and obk_stderr logs contain the stdout and stderr from the script if it does not redirect those outputs elsewhere.

bpubsora: 

This log contains debugging information and execution status for Template Wizard connections to the Oracle instance during database browse and template creation.  If logging into the Wizard as the Oracle user, than the directory must be writable by that user.

user_ops: 

This directory is created automatically and must remain readable and writeable by the Oracle user. The comm and progress files therein are updated by the NetBackup server processes as the backup or restore progresses.  It is always present, and should be gathered for most cases.

bpcd:

This log contains debugging information for the connection attempts to update the comm and progress files.

vnetd: 

This log contains debugging information for the connection attempts from the servers to bpcd and dbclient.

Create the following directories on the media server:

bpbrm: 

The bpbrm process handles backup and restore requests.

bptm: 

The bptm process handles interactions with the tape device. If backing up to a disk storage unit, the bpdm folder should be created.  At NetBackup 6.5 and above, bptm is used for both tape and disk storage units.

Create the following directory on the master server:

bprd: 

The bprd process handles the inbound backup, list, and restore requests from the client server.

bpcompatd: 

May be needed in rare instances if nbjm is unable to connect to the client to update comm files.

nbpem and nbjm may be needed occassionally if the jobs do not go Active in a timely manner for the expected policy.

If this is an RMAN PROXY backup with the Snapshot Client, then include these additional logs:

bpfis: 

From both the backup client and the off-host client.

vnetd and bpcd: 

From the off-host client.

To set the debugging level, go into the Backup, Archive and Restore GUI, and select File | NetBackup Client Properties, and click on the Troubleshooting tab. Increase the General, Verbose, and Database levels to 5. On Unix, the debug level can be changed by editing the bp.conf file and adding the following line:

VERBOSE = 5

If the problem is a performance issue, use VERBOSE = 6 to display delays between dbclient and Oracle.  (This applies only to 5.x and 6.5.4 and above).

In order to ensure the processes pick up the new verbosity settings, it is best to stop and restart all the NetBackup services on the hosts.  These processes are key.  The bprd process can pick up a verbosity change via bprdreq -rereadconfig but requires a restart if the logging directory was created.  The bpbrm and bptm processes do not require a restart unless the job will join a long running MPX group.

On Windows, the Oracle processes must be stopped and restarted for dbclient to pickup up a verbosity change.

Once logging is enabled, run the backup or restore, and collect and send the following information for the Support Engineer to review:

1. Collect the logs from the above directories, making sure to rename the files to reflect the folder from which the log comes, and forward to the Support Engineer

2. In addition, on the Oracle client, send the script file (rename it to .txt so it isn't stripped off by anti-virus software), the Recovery Manager (RMAN) output file, the initSID.ora file, and the zipped contents of the user_ops directory.

3. In the bphdb directory, there should be two additional sets of files, oracle_stdout.<date> and oracle_stderr.<date>. Send these files from the date of the backup or restore attempt.

4. Note the names each of the machines

Additionally, it is extremely helpful to have basic system and configuration information from each of the servers involved in the backup or restore.  For NetBackup 6.x and above, run nbsu on the servers.



To check progress on backup jobs you can use :-


bdbjobs -jobid <id> -U


For restores, you have an option to create the log file when you configure the restore. Look into that particular log file to check progress. Alternatively you can use /usr/openv/netbackup/logs/user_ops/<user name who started the job>/logs


The purpose of this post is to provide a single location for info to be able to set up logs.
The details below are just about the shortest version I can make, that still contains detail/ explanation.

Any NetBackup Engineer should be able to set up logs with the minimum of information, all that 'should be required' for example.

"Please set bptm/ bpbrm on windows media xxx at verbose 5/ general 2 and bpdbm on unix master at versbose 5".