Follow by Email

Saturday, June 21, 2014

Netbackup Restoration Error 147

I have faced an issue during restoration of a unix ( Solaris ) filesystem backup

I have tired the restoration but was not able to restore as it failed with error

“Status 147 - required or specified copy was not found”,

 Later i have examined the bprd logs and i saw the error below

zmasb1:/usr/openv/netbackup/logs/bprd #  more  log.061814 | grep copy #1 not found!

11:27:27.411 [10605] <2> restore files:    proxy_copy = 0

11:27:27.411 [10605] <2> restore files:    alt_rest_copy_num = 0

11:27:27.696 [10605] <2> add_image_to_list: backupid zwftpt1_1278162514 copy #1 not found!

This is issue with the primary image in the netbackup where the PRIMARY_COPY/FRAGMENT mismatch or Corrupted, Now we have to duplicate the image to a good and valid backup image using command bpduplicate. Please keep in mind before you duplicate a image to make it primary image verify the image first. In my case i am going to duplicate last monthly backup image as primary image.


Now go to your NBU console and go to Catalog and there you can search for a valid image which you can restore

Now right clcik on the displayed images and choose verify and check the results, You should see it is successful. Now go back and right click on the image and make it primary copy or use below command to duplicate the image

Tuesday, June 10, 2014

EMC VNX Default Thresholds Limits and SP Cache Behavior when it reaches beyond the water Mark

Today we had a discussion when one of my customer's VNX storage had performance hit and we have started digging down to find out the issue and finally we came to know it is the unique behavior of FLARE as
it has a condition called Forced Flushing. It occurs when the percent count of dirty cache pages crosses over the high watermark and reaches 100%. At that point, the cache starts forcefully flushing unsaved (dirty) data to disk, suspending all host IO. Forced flushing continues until the percent count of dirty pages recedes below the low watermark.

Forced flushing affects the entire array and all workloads served by the array. It significantly increases the host response time until the number of cache dirty pages falls below the low watermark. The Storage Processor gives priority to writing dirty data to disk, rather than allocating new pages for incoming Host IO. The idea of high and low watermark functionality was implemented as a mechanism to avoid forced flushing. The lower the high watermark, the larger the reserved buffer in the cache, and the smaller chance that forced flushing will occur.

So why "the SP Dirty Page% occasionally can reach 95%"? Because there are too many inbound IOs and the backend disks might have been overloaded thus the cache doesn't have enough time to write them to the disks.

Please also find the Default thresholds for different parameters in EMC VNX

160 iops
2500 iops
70 iops
90 iops
160 iops
120 iops
Dirty pages
95 %
Disk resp. time
>15 ms (if total iops > 20)
Average Seek Distance
10 % or > 30 GB/s
Lun response time
>22ms (if total iops > 20)
BE Bandwidth
>320MB/s or 2160MB/s for VNX

Tuesday, June 3, 2014

Moving a Netapp Filer from an old domain to a New Domain

Today i had a situation to change the domain of all my filers due to a major acquisition in my company.

Please note changing domain of a filer will have disruption to you storage accessed through network ( NAS ) make sure No open files at the time OF change cause it may cause corruption to the files but your LUNS will be just fine. Recommended to perform this during off-peak hours.

After change ask users to remount the shares using new fully qualified domain name or can jus use the Filer name followed by share name

Remember before proceeding make sure you have a Windows account with administrative privileges handy.

 First terminate the CIFS
Nayab> cifs terminate

   Then run the cifs setup

Nayab> cifs setup

    Now follow the prompts below and choose

    Do you want to delete the existing filer account information? [no]

    Delete your existing account information by entering yes at the prompt.

    Note: You must delete your existing account information to reach the DNS server entry prompt.

    After deleting your account information, you are given the opportunity to rename the storage system:

    The default name of this filer will be 'Nayab'.
    Do you want to modify this name? [no]:

    Keep the current storage system name by pressing Enter; otherwise, enter yes and enter a new storage system name.

    Data ONTAP displays a list of authentication methods:

    Data ONTAP CIFS services support four styles of user authentication. Choose the one from the list below that best suits your situation.
    (1) Active Directory domain authentication (Active Directory domains only)
    (2) Windows NT 4 domain authentication (Windows NT or Active Directory domains)
    (3) Windows Work group authentication using the filer's local user accounts
    (4) /etc/passwd and/or NIS/LDAP authentication

It chooses the domain 1 by default

 Selection (1-4)? [1]:

Now enter the new domain Name

What is the name of the Active Directory domain? [nayab.corp]: nayabrs.corp

        In Active Directory-based domains, it is essential that the filer's
        time match the domain's internal time so that the Kerberos-based
        authentication system works correctly. If the time difference between
        the filer and the domain controllers is more than 5 minutes,
        authentication will fail. Time services are currently not configured
        on this filer.

Would you like to configure time services? [y]: n

        In order to create an Active Directory machine account for the filer,
        you must supply the name and password of a Windows account with
        sufficient privileges to add computers to the NAYABRS.CORP domain.
Enter the name of the Windows user [Administrator@NAYABRS.CORP]:
Password for Administrator@NAYABRS.CORP:

    Respond to the remainder of the cifs setup prompts; to accept a default value, press Enter.

    Upon exiting, the cifs setup utility starts CIFS.

    Confirm your changes by entering the following command:

    Nayab> cifs domaininfo

cifs domaininfo
NetBios Domain:           NAYAB
Windows 2003 Domain Name: nayab.corp
Type:                     Windows 2003
Filer AD Site:            Singapore

Current Connected DCs:    \\DOMAINC01
Total DC addresses found: 4
Preferred Addresses:
Favored Addresses:
                    DOMAINC01         PDC
Other Addresses:

Connected AD LDAP Server: \\domainc02.nayab.corp
Preferred Addresses:
Favored Addresses:
Other Addresses: