Prerequisites for GRT Backup/Restore NetBackup 7.5 and Windows 2008 R2 Domain Controller

Pre-Requisites

NFS must be installed on Media Server and All AD Domain Controllers which are to be backed up with GRT enabled

Install NFS Services on Media Server and All the Active Domain Controllers Windows 2008/R2

For NFS, Add File Service Server Role

Select Services for Network File System as Role Service , Complete the wizard

From Media Server configure portmap service to start automatically after server restart

From CMD execute “sc config portmap start= auto”

From Service.msc console Stop and Disable below services

Server for NFS on both Active Domain Controller and Media Server And Disable Client for NFS on Media Server

If Active Domain controller is also Media Server disable both Client/Server for NFS Service

Create No.Restrictions Touch (empty) File on Media Server under /veritas/netbackup/db/altnames

If altnames directory does not exist, create it

Make sure Touch file does not have suffix of .txt it should be No.Restrictions only

To rename extension of file, from windows explorer navigate to Folder Option (Alt+T+O)

Folder options=> View => Uncheck Hide Extensions for known file types => Apply => Ok

Make sure you revoke above change after file is created.

Privileged Account for GRT

NetBackup Client Services at Active Domain Controller must be running with Domain Admin Privileged Account

How to identify the disk name in Netapp

We all pretty well know every disk will have a universal unique identifier UUID or serial number , but  each disk will have a unique name depending on how they are connected to the storage system.

Note :- Always remember For internal disks, the slot number is zero, and the internal port number depends on the system model.

First knowing the basics each disk shelf will have bay's to hold the disk

Ex:- A Netapp DS4243 is a 24 bay disk shelf which is 4U in size.

Now each disk shelf will be identified with a name based on the Shelf ,Bay and the port to which the shelf is connected and we have different methodology for different type of DISK CONNECTION
                 
For the SAS, Direct Attached type of disk connection the disk name will  be based on

<slot><port>.<shelf ID>.<bay>  ( In short i remember it as SPSB ) 


Ex:- Which means say if i have a disk in bay 12 of Shelf 1 which is connected to onboard port A

My disk ID would be :- 0a.1.12 

For the FC-AL, direct-attached type of disk connection the name is based on

<slot><port>.<loopID> ( In short i remember it as SPL )

Ex:- If my disk loop ID is 20 connected to expansion card on slot 8 to port C

My disk ID would be :- 8c.20

For FC-AL , Switch attached type of connection the name is based as below

<switch_name>.<switch_port>.<loopID>


Ex:- If my disk loop ID is 40 connected to port 4 of a switch 4

My Disk ID would be :- SW4.4.40








  

Disk Physical Size Vs Usable Size

Tool to analyze Netapp Perfstat Output

Wondering how to analyze Netapp Perfstat , The tool below will help you out to know the CPU , Disk Overhead etc...

Download Link :- http://www.everythingvm.com/sites/default/files/SCPerfstatAnalyzer.exe

1) Download and run the .exe from the link above 

2) It will open a your command prompt in windows machine to key in the Perfstat file name 

3) Enter the perfstat file name with complete location

 4) Now enter the controller name for which you want to analyze the perfstat

This will generate three text files for NETWORK , CPU & DISK in the same folder where your perfstat_analyzer.exe was present 

Difference Between Netbackup and Backupexec

Well we know Netbackup is for Enterprise level and Backup is for Mid-sized environments , I would like to highlight few more difference's which are mainly considered while discussing about these two leading backup products from Symantec.

  

Category	Backup_Exec	Netbackup
Latest Version	2014	7.6.0.2
Database in the backend	Microsoft SQL	Sybase
Tape Format	BKF	TAR
Reporting	Native	OpsCenter
Management	CASO (Central Admin Server Option) can manage multiple media server deployed across Domain.	NetBackup Master Server, can manage multiple media server, SAN media servers and clients centrally.
Can read from NetBackup/Backup Exec	Not supported to read NetBackup images to Backup Exec	Supported until BE 2012 , later versions not supported
NDMP	Supported	Supported
De-Duplication	Supported	Supported
GRT	Supported	Supported
BMR	Supported	Supported
Tape Library, VTL and autoloader	Supported	Supported
Tape-out protocols FC & ISCSi	Supported	Supported
Multistreaming backups	Not Supported	Supported
Multiplexed backups	Not Supported	Supported
Vmware and Hyper-V Environments	Supported	Supported
Offline and Online backups	Supported	Supported
Automatic scheduled and Manual Backups	Supported	Supported
Media Server Support	Windows Only	Windows as well as UNIX
Client Support	Windows and Linux Only	Windows as well as UNIX
Disk Based Backups	Supported	Supported
Encryption	Supported	Supported
SAN backups	Supported	Supported
Convert backups to virtual machines	Supported	Supported

Difference between Snapmirror and Snapvault

Most of my clients had this doubt about Snapmirror and Snapvault

1) What are the differences between these two as both these perform a copy from source to destination

2) why we need to have two products for backup

3) Why do i need to buy two product license , Instead i can just have one product

4) What is my RTO and RPO with these products and which one is better for DR

Here i am trying to tell the exact difference between Snapmirror and Snapvault, First going in to basics

What is Snapvault ?

A SnapVault backup is a collection of Snapshot copies on a Flex volume that you can restore data from if the primary data is not usable. Snapshot copies are created based on a Snapshot policy. The SnapVault backup backs up Snapshot copies based on its schedule and SnapVault policy rules.

A SnapVault backup is a disk-to-disk backup solution that you can also use to offload tape backups. In the event of data loss or corruption on a system, backed-up data can be restored from the SnapVault secondary volume with less downtime and uncertainty than is associated with conventional tape backup and restore operations.

What is Snapmirror ?

SnapMirror is a feature of Data ONTAP that enables you to replicate data. SnapMirror enables you to replicate data from specified source volumes or qtrees to specified destination volumes or qtrees, respectively. You need a separate license to use SnapMirror.

You can use SnapMirror to replicate data within the same storage system or with different storage systems.

Now we see what is the difference between Snapvault and Snapmirror

First statement i would say the difference is "Snapvault is a backup solution where snapmirror is a DR solution"

Snapvault is a backup solution where we can have long snapshot retention periods on the destination filer and slower disks can be used at the destination side with low RPM's to minimize the budget , In case of disaster occurrence we can restore data from destination filer to source filer, But we cannot make the destination as source to serve the data as Snapvault destinations are READ ONLY.

Snapmirror is a DR solution where we can use Sync and Semi-Sync , Async relationships  and also we can easily restore the accidentally deleted, or lost data to source filer, if there are no updates were performed meanwhile. If there is a total disaster on the source side we can immediately perform a reverse snapmirror in case of total disaster of source we can make the destination volume/qtree as read-write and provide access to the clients. Which means low RTO and RPO , Which means low outages. Once the Source is ready we can resync the destination to source and continue with source as before.

Notable difference

Qtree SnapMirror

More suitable for providing immediate failover capability.

Uses the same functionality and licensing on the source and destination systems.

Transfers can be scheduled at a maximum rate of once every minute.

Relationships can be reversed. This allows the source to be re-synchronized with changes made at the destination.

Snapvault

More suitable where data availability is less critical, and immediate failover is not required.

Uses SnapVault source system and SnapVault destination system, which provide different functionality.

Transfers can be scheduled at a maximum rate of once every hour.

Snapshot copies are retained and deleted on a specified schedule.

Relationships cannot be reversed. It provides the capability to transfer data from the destination to the source only to restore data. The direction of replication cannot be reversed.

To Summarize 

Snapvault is moreover a Backup solution rather than a disaster recovery solution but imagine in case if you are  in deep trouble and need to have your production up asap then go for converting Snapvault qtree in to a Snapmirror qtree.( I haven't tried but it can be done from DIAG mode with Snapvault convert command ).

Snapmirror is purely replication solution which saves us in case of disaster.

So based on differences above we can easily judge which solution will have more RTO & RPO.

License structure was designed by Netapp , As we already know we need to have primary as well as secondary license.

Hope this helped !!

Setting up Netapp Passwordless Login

Usually we configure the passwordless login for Netapp filer by mounting or mapping Vol0 through NFS or CIFS , If in case your environment don't have these licenses then you can try the procedure below even if you have license i prefer the below one as its easy. As you jus need to unlock the diag account and proceed.

If you do not have neither CIFS nor NFS licenses, you could create this directory using diag account.

Note: take care using this account.

First, enter in advanced mode:
filer> priv set advanced

Now, unlock and set a password to diag account:

filer*> useradmin diaguser unlock

filer*> useradmin diaguser password ( Set a password )

Enter in the systemshell, create the directory you need and put the pubkey generated in the authorized_keys file:

filer*> systemshell

login: diag

Password: the same you set in the previous step above

filer% mkdir -p /mroot/etc/sshd/root/.ssh

filer% vi /mroot/etc/sshd/root/.ssh/authorized_keys

Now Copy your server SSH keys here and Save the file

filer% sudo chown -R root:wheel /mroot/etc/sshd/root

filer% sudo chmod -R 0600 /mroot/etc/sshd/root

Then, exit systemshell, lock diag account and exit advanced mode:

filer% exit

Lock the Diag User back

filer*> useradmin diaguser lock

filer*> priv set admin

Netapp Data Ontap Clustered Infinite volume Limitations

Netbackup Restoration Error 147

I have faced an issue during restoration of a unix ( Solaris ) filesystem backup

I have tired the restoration but was not able to restore as it failed with error

“Status 147 - required or specified copy was not found”,

 Later i have examined the bprd logs and i saw the error below

zmasb1:/usr/openv/netbackup/logs/bprd #  more  log.061814 | grep copy #1 not found!

11:27:27.411 [10605] <2> restore files:    proxy_copy = 0

11:27:27.411 [10605] <2> restore files:    alt_rest_copy_num = 0

11:27:27.696 [10605] <2> add_image_to_list: backupid zwftpt1_1278162514 copy #1 not found!

This is issue with the primary image in the netbackup where the PRIMARY_COPY/FRAGMENT mismatch or Corrupted, Now we have to duplicate the image to a good and valid backup image using command bpduplicate. Please keep in mind before you duplicate a image to make it primary image verify the image first. In my case i am going to duplicate last monthly backup image as primary image.

Verification..

Now go to your NBU console and go to Catalog and there you can search for a valid image which you can restore

Now right clcik on the displayed images and choose verify and check the results, You should see it is successful. Now go back and right click on the image and make it primary copy or use below command to duplicate the image

EMC VNX Default Thresholds Limits and SP Cache Behavior when it reaches beyond the water Mark

Today we had a discussion when one of my customer's VNX storage had performance hit and we have started digging down to find out the issue and finally we came to know it is the unique behavior of FLARE as
it has a condition called Forced Flushing. It occurs when the percent count of dirty cache pages crosses over the high watermark and reaches 100%. At that point, the cache starts forcefully flushing unsaved (dirty) data to disk, suspending all host IO. Forced flushing continues until the percent count of dirty pages recedes below the low watermark.

Forced flushing affects the entire array and all workloads served by the array. It significantly increases the host response time until the number of cache dirty pages falls below the low watermark. The Storage Processor gives priority to writing dirty data to disk, rather than allocating new pages for incoming Host IO. The idea of high and low watermark functionality was implemented as a mechanism to avoid forced flushing. The lower the high watermark, the larger the reserved buffer in the cache, and the smaller chance that forced flushing will occur.

So why "the SP Dirty Page% occasionally can reach 95%"? Because there are too many inbound IOs and the backend disks might have been overloaded thus the cache doesn't have enough time to write them to the disks.

Please also find the Default thresholds for different parameters in EMC VNX

FC	160 iops
EFD	2500 iops
ATA	70 iops
SATAII	90 iops
SAS	160 iops
NL SAS	120 iops
Dirty pages	95 %
Disk resp. time	>15 ms (if total iops > 20)
Average Seek Distance	10 % or > 30 GB/s
Lun response time	>22ms (if total iops > 20)
BE Bandwidth	>320MB/s or 2160MB/s for VNX