Tool to analyze Netapp Perfstat Output

Wondering how to analyze Netapp Perfstat , The tool below will help you out to know the CPU , Disk Overhead etc...

Download Link :- http://www.everythingvm.com/sites/default/files/SCPerfstatAnalyzer.exe

1) Download and run the .exe from the link above 

2) It will open a your command prompt in windows machine to key in the Perfstat file name 

3) Enter the perfstat file name with complete location

 4) Now enter the controller name for which you want to analyze the perfstat

This will generate three text files for NETWORK , CPU & DISK in the same folder where your perfstat_analyzer.exe was present 

Difference Between Netbackup and Backupexec

Well we know Netbackup is for Enterprise level and Backup is for Mid-sized environments , I would like to highlight few more difference's which are mainly considered while discussing about these two leading backup products from Symantec.

  

Category	Backup_Exec	Netbackup
Latest Version	2014	7.6.0.2
Database in the backend	Microsoft SQL	Sybase
Tape Format	BKF	TAR
Reporting	Native	OpsCenter
Management	CASO (Central Admin Server Option) can manage multiple media server deployed across Domain.	NetBackup Master Server, can manage multiple media server, SAN media servers and clients centrally.
Can read from NetBackup/Backup Exec	Not supported to read NetBackup images to Backup Exec	Supported until BE 2012 , later versions not supported
NDMP	Supported	Supported
De-Duplication	Supported	Supported
GRT	Supported	Supported
BMR	Supported	Supported
Tape Library, VTL and autoloader	Supported	Supported
Tape-out protocols FC & ISCSi	Supported	Supported
Multistreaming backups	Not Supported	Supported
Multiplexed backups	Not Supported	Supported
Vmware and Hyper-V Environments	Supported	Supported
Offline and Online backups	Supported	Supported
Automatic scheduled and Manual Backups	Supported	Supported
Media Server Support	Windows Only	Windows as well as UNIX
Client Support	Windows and Linux Only	Windows as well as UNIX
Disk Based Backups	Supported	Supported
Encryption	Supported	Supported
SAN backups	Supported	Supported
Convert backups to virtual machines	Supported	Supported

Difference between Snapmirror and Snapvault

Most of my clients had this doubt about Snapmirror and Snapvault

1) What are the differences between these two as both these perform a copy from source to destination

2) why we need to have two products for backup

3) Why do i need to buy two product license , Instead i can just have one product

4) What is my RTO and RPO with these products and which one is better for DR

Here i am trying to tell the exact difference between Snapmirror and Snapvault, First going in to basics

What is Snapvault ?

A SnapVault backup is a collection of Snapshot copies on a Flex volume that you can restore data from if the primary data is not usable. Snapshot copies are created based on a Snapshot policy. The SnapVault backup backs up Snapshot copies based on its schedule and SnapVault policy rules.

A SnapVault backup is a disk-to-disk backup solution that you can also use to offload tape backups. In the event of data loss or corruption on a system, backed-up data can be restored from the SnapVault secondary volume with less downtime and uncertainty than is associated with conventional tape backup and restore operations.

What is Snapmirror ?

SnapMirror is a feature of Data ONTAP that enables you to replicate data. SnapMirror enables you to replicate data from specified source volumes or qtrees to specified destination volumes or qtrees, respectively. You need a separate license to use SnapMirror.

You can use SnapMirror to replicate data within the same storage system or with different storage systems.

Now we see what is the difference between Snapvault and Snapmirror

First statement i would say the difference is "Snapvault is a backup solution where snapmirror is a DR solution"

Snapvault is a backup solution where we can have long snapshot retention periods on the destination filer and slower disks can be used at the destination side with low RPM's to minimize the budget , In case of disaster occurrence we can restore data from destination filer to source filer, But we cannot make the destination as source to serve the data as Snapvault destinations are READ ONLY.

Snapmirror is a DR solution where we can use Sync and Semi-Sync , Async relationships  and also we can easily restore the accidentally deleted, or lost data to source filer, if there are no updates were performed meanwhile. If there is a total disaster on the source side we can immediately perform a reverse snapmirror in case of total disaster of source we can make the destination volume/qtree as read-write and provide access to the clients. Which means low RTO and RPO , Which means low outages. Once the Source is ready we can resync the destination to source and continue with source as before.

Notable difference

Qtree SnapMirror

More suitable for providing immediate failover capability.

Uses the same functionality and licensing on the source and destination systems.

Transfers can be scheduled at a maximum rate of once every minute.

Relationships can be reversed. This allows the source to be re-synchronized with changes made at the destination.

Snapvault

More suitable where data availability is less critical, and immediate failover is not required.

Uses SnapVault source system and SnapVault destination system, which provide different functionality.

Transfers can be scheduled at a maximum rate of once every hour.

Snapshot copies are retained and deleted on a specified schedule.

Relationships cannot be reversed. It provides the capability to transfer data from the destination to the source only to restore data. The direction of replication cannot be reversed.

To Summarize 

Snapvault is moreover a Backup solution rather than a disaster recovery solution but imagine in case if you are  in deep trouble and need to have your production up asap then go for converting Snapvault qtree in to a Snapmirror qtree.( I haven't tried but it can be done from DIAG mode with Snapvault convert command ).

Snapmirror is purely replication solution which saves us in case of disaster.

So based on differences above we can easily judge which solution will have more RTO & RPO.

License structure was designed by Netapp , As we already know we need to have primary as well as secondary license.

Hope this helped !!

Setting up Netapp Passwordless Login

Usually we configure the passwordless login for Netapp filer by mounting or mapping Vol0 through NFS or CIFS , If in case your environment don't have these licenses then you can try the procedure below even if you have license i prefer the below one as its easy. As you jus need to unlock the diag account and proceed.

If you do not have neither CIFS nor NFS licenses, you could create this directory using diag account.

Note: take care using this account.

First, enter in advanced mode:
filer> priv set advanced

Now, unlock and set a password to diag account:

filer*> useradmin diaguser unlock

filer*> useradmin diaguser password ( Set a password )

Enter in the systemshell, create the directory you need and put the pubkey generated in the authorized_keys file:

filer*> systemshell

login: diag

Password: the same you set in the previous step above

filer% mkdir -p /mroot/etc/sshd/root/.ssh

filer% vi /mroot/etc/sshd/root/.ssh/authorized_keys

Now Copy your server SSH keys here and Save the file

filer% sudo chown -R root:wheel /mroot/etc/sshd/root

filer% sudo chmod -R 0600 /mroot/etc/sshd/root

Then, exit systemshell, lock diag account and exit advanced mode:

filer% exit

Lock the Diag User back

filer*> useradmin diaguser lock

filer*> priv set admin

Netapp Data Ontap Clustered Infinite volume Limitations

Netbackup Restoration Error 147

I have faced an issue during restoration of a unix ( Solaris ) filesystem backup

I have tired the restoration but was not able to restore as it failed with error

“Status 147 - required or specified copy was not found”,

 Later i have examined the bprd logs and i saw the error below

zmasb1:/usr/openv/netbackup/logs/bprd #  more  log.061814 | grep copy #1 not found!

11:27:27.411 [10605] <2> restore files:    proxy_copy = 0

11:27:27.411 [10605] <2> restore files:    alt_rest_copy_num = 0

11:27:27.696 [10605] <2> add_image_to_list: backupid zwftpt1_1278162514 copy #1 not found!

This is issue with the primary image in the netbackup where the PRIMARY_COPY/FRAGMENT mismatch or Corrupted, Now we have to duplicate the image to a good and valid backup image using command bpduplicate. Please keep in mind before you duplicate a image to make it primary image verify the image first. In my case i am going to duplicate last monthly backup image as primary image.

Verification..

Now go to your NBU console and go to Catalog and there you can search for a valid image which you can restore

Now right clcik on the displayed images and choose verify and check the results, You should see it is successful. Now go back and right click on the image and make it primary copy or use below command to duplicate the image

EMC VNX Default Thresholds Limits and SP Cache Behavior when it reaches beyond the water Mark

Today we had a discussion when one of my customer's VNX storage had performance hit and we have started digging down to find out the issue and finally we came to know it is the unique behavior of FLARE as
it has a condition called Forced Flushing. It occurs when the percent count of dirty cache pages crosses over the high watermark and reaches 100%. At that point, the cache starts forcefully flushing unsaved (dirty) data to disk, suspending all host IO. Forced flushing continues until the percent count of dirty pages recedes below the low watermark.

Forced flushing affects the entire array and all workloads served by the array. It significantly increases the host response time until the number of cache dirty pages falls below the low watermark. The Storage Processor gives priority to writing dirty data to disk, rather than allocating new pages for incoming Host IO. The idea of high and low watermark functionality was implemented as a mechanism to avoid forced flushing. The lower the high watermark, the larger the reserved buffer in the cache, and the smaller chance that forced flushing will occur.

So why "the SP Dirty Page% occasionally can reach 95%"? Because there are too many inbound IOs and the backend disks might have been overloaded thus the cache doesn't have enough time to write them to the disks.

Please also find the Default thresholds for different parameters in EMC VNX

FC	160 iops
EFD	2500 iops
ATA	70 iops
SATAII	90 iops
SAS	160 iops
NL SAS	120 iops
Dirty pages	95 %
Disk resp. time	>15 ms (if total iops > 20)
Average Seek Distance	10 % or > 30 GB/s
Lun response time	>22ms (if total iops > 20)
BE Bandwidth	>320MB/s or 2160MB/s for VNX

Moving a Netapp Filer from an old domain to a New Domain

Today i had a situation to change the domain of all my filers due to a major acquisition in my company.

Please note changing domain of a filer will have disruption to you storage accessed through network ( NAS ) make sure No open files at the time OF change cause it may cause corruption to the files but your LUNS will be just fine. Recommended to perform this during off-peak hours.

After change ask users to remount the shares using new fully qualified domain name or can jus use the Filer name followed by share name

Remember before proceeding make sure you have a Windows account with administrative privileges handy.

 First terminate the CIFS
   
Nayab> cifs terminate

   Then run the cifs setup

Nayab> cifs setup

    Now follow the prompts below and choose

    Do you want to delete the existing filer account information? [no]

    Delete your existing account information by entering yes at the prompt.

    Note: You must delete your existing account information to reach the DNS server entry prompt.

    After deleting your account information, you are given the opportunity to rename the storage system:

    The default name of this filer will be 'Nayab'.
    Do you want to modify this name? [no]:

    Keep the current storage system name by pressing Enter; otherwise, enter yes and enter a new storage system name.

    Data ONTAP displays a list of authentication methods:

    Data ONTAP CIFS services support four styles of user authentication. Choose the one from the list below that best suits your situation.
    (1) Active Directory domain authentication (Active Directory domains only)
    (2) Windows NT 4 domain authentication (Windows NT or Active Directory domains)
    (3) Windows Work group authentication using the filer's local user accounts
    (4) /etc/passwd and/or NIS/LDAP authentication

It chooses the domain 1 by default

 Selection (1-4)? [1]:

Now enter the new domain Name

What is the name of the Active Directory domain? [nayab.corp]: nayabrs.corp

        In Active Directory-based domains, it is essential that the filer's
        time match the domain's internal time so that the Kerberos-based
        authentication system works correctly. If the time difference between
        the filer and the domain controllers is more than 5 minutes,
        authentication will fail. Time services are currently not configured
        on this filer.

Would you like to configure time services? [y]: n

        In order to create an Active Directory machine account for the filer,
        you must supply the name and password of a Windows account with
        sufficient privileges to add computers to the NAYABRS.CORP domain.
Enter the name of the Windows user [Administrator@NAYABRS.CORP]:
Password for Administrator@NAYABRS.CORP:

    Respond to the remainder of the cifs setup prompts; to accept a default value, press Enter.

    Upon exiting, the cifs setup utility starts CIFS.

    Confirm your changes by entering the following command:

    Nayab> cifs domaininfo



cifs domaininfo
NetBios Domain:           NAYAB
Windows 2003 Domain Name: nayab.corp
Type:                     Windows 2003
Filer AD Site:            Singapore

Current Connected DCs:    \\DOMAINC01
Total DC addresses found: 4
Preferred Addresses:
                          None
Favored Addresses:
                          192.168.2.34    DOMAINC01         PDC
                          192..168.3.35
                           92.168.2.20                  PDC
                                               PDC
Other Addresses:
                          192.254.52.71                    BDC

Connected AD LDAP Server: \\domainc02.nayab.corp
Preferred Addresses:
                          None
Favored Addresses:
                          192.168.2.34
                           domain02.nayab.corp
                          192..168.3.35
                           domainc02.nayab.corp
                          192.168.2.20
                           domainc01.nayab.corp
Other Addresses:
                          None