Netapp Snap Restore at Volume Level

Today i have performed snap restore on few production volume reverting to the snapshot of dated 20-May-2014. Here i share the procedure with you all.

I have restored snapshot of all 12 prod volumes. Please be noted snap restore can be performed both at VOLUME LEVEL and FILE LEVEL.

Volume level is to revert back the volume to whatever the date you want based on the snapshot in my case my latest snapshot is on 20-May-2014 so I’m  reverting back to it

File level is to restore a single file , we have to mention  type   –t  (vol | file )  while restore.

Im doing at volume level....

Example:-


snap restore -t vol  <vol-name>

snap restore -t vol snap_prodcli_prod_fb

WARNING! This will revert the volume to a previous snapshot.

All modifications to the volume after the snapshot will be

irrevocably lost.

Volume snap_prodcli_prod_fb will be made restricted briefly before coming back online.

Are you sure you want to do this? yes

The following snapshots are available for volume snap_prodcli_prod_fb:

     date            name
------------    ---------

May 20 17:12    20140520_bkup  -->  I am Reverting to this Snapshot

May 12 05:50    20140512_bkup

May 09 18:35    filervltp1(0151751825)_snap_prodcli_prod_fb.1

May 09 17:55    20140509_coldbkup

Apr 10 15:41    20140410_coldbkup

Which snapshot would you like to revert volumesnap_prodcli_prod_fb to?  20140520_bkup   ( Here mention the snapshot name to which you want to revert )

You have selected snap_prodcli_prod_fb,  snapshot 20140520_bkup

Proceed with revert? yes

Thu May 22 10:58:39 SGT [filervltp1:wafl.snaprestore.revert:notice]: Reverting volume snap_prodcli_prod_fb to a previous snapshot.

Volume snap_prodcli_prod_fb : revert successful.

It’s been successful :)

Restoring Oracle Archive Logs using RMAN from Tapes

I got a request from my client asking to restore Archivelogs of  particular sequence. let me brief you all our Oracle using ASM and our backup software is Netbackup.

Generally you can restore archive logs from the OS backup by starting the BAR utility from Netbackup MASTER SERVER but in my case these are ASM files so it cannot be seen from the OS backups so have to invoke the BAR from the CLIENT but i faced an error when i tried to list the files in BAR " Library Binaries Error" something strange so i have opened a case with Netbackup but Netbackup denied support as my NBU version is 6.5.4 which is EOS ( End of Support ) already , So now i have only one way to restore is through RMAN , After some struggle with google  i was able to build a script to restore my archivelogs



Note:- I am going to restore the archivelogs to an alternate location on the same server may be /tmp in my case but please check if you have enough space in the particular directory before proceeding with the restore and also get the LOGSEQUENCE  from your DBA

Login to the Host then switch to oracle

prdbclient:~ #  su - oracle

prdbclient: : oracle:>

Now go to the instance for which you want to restore Archivelogs

 prdbclient: : oracle:> prdb1



 prdbclient:prdb1: oracle:>



Once you are logged in to the instance now go to the RECOVERY MANAGER ( RMAN )   PROMPT ( Check with your DBA for the home directory of Oracle to be set )

/OraBase/V10010/bin -->  Is my Oracle Home directory

prdb1/prdb1 -->  Oracle Userid and Password ( You can get userID and Password from you RMAN SCRIPT )

Now type in as below to go to RMAN prompt of instance prdb1

prdbclient:prdb1: oracle:> /OraBase/V10010/bin/rman target / catalog prdb1/prdb1@rman

Once done run the restoration script

RMAN>

 run {

set archivelog destination to '/tmp/PRDB1_Archlogs';

ALLOCATE CHANNEL t1 TYPE 'SBT_TAPE';

SEND 'NB_ORA_CLIENT=prdbclient, NB_ORA_SERV=masbak, NB_ORA_POLICY=prdbclient_prdb1_archlog_daily';

restore archivelog from logseq=443196 until logseq=433299;

RELEASE CHANNEL t1;

}


Monitor the Restoration Job from the Netbackup Console

 Once completed successfully let the DBA do his job :)

Netapp Not able to access CIFS shares

 We May not be able to access CIFS shares ( including C$ and ETC$ ) on a New Filer I have come across this issue and i tried all possible ways to figure out the cause and finally i was able to fix the issue, 

The reason is when you run CIFS SETUP on your filer it will create a user called "PCUSER"  ( All windows users will be mapped to this users while accessing the shares on the filer )  , you can check it from the below command your file should be similar to below

FAS3220> rdfile  /etc/passwd

root : : 0 : : : / :
pcuser : : 65534 : 65534 : : / :
nobody : : 65534 : 65534 : : / :
ftp : : 65533 : 65533 : FTP Anonymous : /home/ftp :


For some reason  it does not create the user pcuser and i have noticed the /etc/passwd was empty so all what i did is to copy the /etc/passwd from one of my old filers and don't forget to run source command after copying

FAS3220> source  /etc/passwd

Once done you should be able to access you shares

Happy Knowledge Sharing :) 

Migrating a root Volume in Netapp

  Please remember that moving a root volume requires a reboot of the controller.  
Therefore, i suggest to complete this activity in your Maintenance windows. 

It is always good to have a snapshot created before starting any migrations 


FASProd>snap create -V vol0 vol0_snap


 1.       Disable the cluster:  

FASProd>cf disable

2.       Check the size of current vol0:  

FASProd>df -Vh  vol0

3.       Create a new root volume on destination aggr:    

FASProd>vol create  vol0_new  dest_aggr  <SIZE>

4.       Copy the data to the new volume:

FASProd> ndmpcopy /vol/vol0  /vol/vol0_new ( Can also use Vol Copy )

 5.       Once copy is done now Rename the old root volume: 

FASProd>vol rename vol0 vol0_old

6.       Rename the new root volume:  

FASProd>vol rename vol0_new vol0

7.       Now change the new vol0 to be used as root volume 

FASProd> vol options vol0 root

8.       Reboot the controller:  

FASProd>reboot

9.       Confirm the destination aggr now hosts the root vol0:  

FASProd>vol status vol0  

10.   Once confirmed proceed to Offline and destroy the old volume

FASProd> vol offline vol0_old  

FASProd> vol destroy vol0_old

Now i am a NTSP ( Nimble Technical Sales Professional )

Knowing what is Vmware Vcenter, ESXi, Vsphere

I was really confused with the terms Vsphere , Vcenter and ESXi and i have thought to write up the difference so that it might help someone at some point of time

What is vSphere ?

vSphere is suite of products packaged shipped by VMware virtualization company, which includes products. Like VMware ESXi hypervisor, VMware vCenter Server, etc.

VMware ESXi ?

ESXi or hypervisor is bare metal OS which virtualizes x86 server hardware. You use ESXi ( OS ) installation  media to install OS on top of hardware just like you install any other OS like Windows or Linux

Once you have VMware ESXi installed on a x86 server hardware I is ready to virtualized. What does it mean ? - It means you can now use same server to create , configure, run more than one virtual machines and install required Operating systems within Virtual Machines they are referred as Guest Operating systems which could be Windows or Linux etc

 What is vCenter ?

 

Vcenter is another Vmware product used to manage one or more ESXi servers comes in the Vsphere software package.

Netbackup Vmoprcmd Cobra Communication Error

If you have your Netbackup Master Server Version 7.6 and your media server ( We had 7.0.1 ) version lower than master version you may face the error while you run the command Vmoprcmd 

 Even the compatibility matrix says 7.6 is compatible with 7.0.1 but still we have faced the below error as soon as we upgraded the master server from 7.0.1 to 7.6.0.1,

Cobra Communication Error 

Solution :-  Is to Upgrade Media server also from 7.0.1 to 7.6.0.1 to fix this issue.

Netbackup Oracle backup Error 6 On HP-UX Client

One of our Oracle DB backups were failing with error 6 which is a generic error for DB backups using Netbackup and from the error log i have found

Errors in file /OraBase/admin/Test50/bdump/Test50_j000_29761.trc:

ORA-12012: error on auto execute of job 661138

ORA-00257: archiver error. Connect internal only, until freed.




Error:- ORA-00257: archiver error. Connect internal only, until freed.

Solution:- 

And i see this is moreover Oracle Side to troubleshoot rather the Netbackup side and after a small research i have found the solution is to move some of the archive logs 

--> find the location of Archive destination by
 show parameter archive_dest
 
 lets say it provide  LOCATION=/Test50/oradata/mydb/arch

-->  move some files to some other location using os command
 cd /Test50/oradata/mydb/arch
 mv /Test50/oradata/mydb/arch/* /Test5/oradata/mydb/arch-bkp/
 
If you want to do it from RMAN , try as below.

rman target /

RMAN> backup archive log all format '/Test50/oradata/mydb/arch-bkp';

RMAN> delete archive until time 'trunc(sysdate)';

Error:- ORA-12012: error on auto execute of job 7863452

Solution:- 

The underlying table required as part of one of the schedule maintenance tasks has invalid values. 
The related table is Test50.Testing_Base. 

After taking a look at the job log we see the failed execution attempts:

SQL> select log_date,status from dba_scheduler_job_run_details 
     where job_name='Test50_DBTesting_JOB';

LOG_DATE                              STATUS
------------------------------------- ------------------------------
11-FEB-14 10.00.07.3668 AM -06:00    FAILED
04-FEB-14 08.00.05.59665 AM -06:00   FAILED


2 rows selected.

Solution

Check the invalid values on the table and remove them.

SQL> select * from Test50.Testing_Base;

      DBID INSTANCE_NAME    BASELINE_ID BSLN_GUID                        TI A STATUS           LAST_COMPUT
---------- ---------------- ----------- -------------------------------- -- - ---------------- -----------
4056791929  orcl                       0 24T6YH78JNSHDT89ONSNS09Q2NND8728 NW N ACTIVE           140917:2200
4052355478  ORCL3                      0 12MDNSIdGAJKW474IETJBESGJH539930 ND Y ACTIVE           141022:2000


As we can see the above orcl is corrupted , Now we proceed to delete it.


The corrupt row can be removed as below.


SQL> DELETE FROM Test50.Testing_Base WHERE INSTANCE_NAME ='orcl';

1 row deleted.

SQL> commit;

Commit complete.

Manually re-execute the job and check the execution log, it must show the job executed successfully.  It takes a couple of minutes after execution to show the results in the log table.

SQL> exec dbms_scheduler.run_job('Test50.Testing_Base',false);
PL/SQL procedure successfully completed.


The issue was fixed, validate it by querying

SQL> select log_date,status from dba_scheduler_job_run_details 
     where job_name='Test50.Testing_Base';

LOG_DATE                              STATUS
------------------------------------- ------------------------------
14-FEB-14 11.00.07.315077 AM -06:30   FAILED
15-FEB-14 11.00.05.595559 AM -06:30   FAILED
17-FEB-14 03.41.20.714453 Am -06:30   SUCCEEDED

Now try re-run the job it should be successful :)

Netapp Volume DataMotion Highlights

Today i have performed Netapp DataMotion for volumes to migrate volumes from One Aggregate to another aggregate and i can see DataMotion uses SNAPMIRROR in the back end to move the data from OLD VOLUME to the NEW VOLUME. Please find the brief summary below how it works...

Note :- My volumes has ORACLE DB running on them and i have seen no disruption to the application during DataMotion it went smooth and Hassle Free.

prodfiler4> vol move start production_vol_adm ataggr0 -k

Use " -k " option if you want to keep the source volume as once after the VOL MOVE , Netapp destroys the source volume as all the new reads and writes were directed to the New Volume ( Destination Volume)

prodfiler4> Wed Feb 26 16:14:50 SGT [prodfiler4:vol.move.Start:info]: Move of volume production_vol_adm to aggr ataggr0 started
Creation of volume 'ndm_dstvol_1393402490' with size 21474836480  on containing aggregate
'ataggr0' has completed.

Volume 'ndm_dstvol_1393402490' is now restricted.

Wed Feb 26 16:15:13 SGT [prodfiler4:vol.move.transferStart:info]: Baseline transfer from volume production_vol_adm to ndm_dstvol_1393402490 started.
Transfer started.

Monitor progress with 'snapmirror status' or the snapmirror log.

Wed Feb 26 16:16:54 SGT prodfiler4:vol.move.transferStatus:info]: Baseline transfer from volume production_vol_adm to ndm_dstvol_1393402490 took 97 secs and transferred 1935528 KB data.

Wed Feb 26 16:16:56 SGT [prodfiler4:vol.move.transferStart:info]: Update from volume production_vol_adm to ndm_dstvol_1393402490 started.
Transfer started.

Monitor progress with 'snapmirror status' or the snapmirror log.

Wed Feb 26 16:17:18 SGT [prodfiler4:vol.move.transferStatus:info]: Update from volume production_vol_adm to ndm_dstvol_1393402490 took 13 secs and transferred 1160 KB data.

Wed Feb 26 16:17:23 SGT [prodfiler4:vol.move.transferStart:info]: Update from volume production_vol_adm to ndm_dstvol_1393402490 started.
Transfer started.

Monitor progress with 'snapmirror status' or the snapmirror log.

Wed Feb 26 16:17:44 SGT [prodfiler4:vol.move.transferStatus:info]: Update from volume production_vol_adm to ndm_dstvol_1393402490 took 12 secs and transferred 1104 KB data.

Wed Feb 26 16:17:44 SGT [prodfiler4:vol.move.updateTimePrediction:info]: Expected time for next update from volume production_vol_adm to ndm_dstvol_1393402490 is 12 secs to transfer 272 KB data.

Wed Feb 26 16:17:52 SGT [prodfiler4:vol.move.cutoverStart:info]: Cutover started for vol move of volume production_vol_adm to aggr ataggr0.
Transfer started.

Monitor progress with 'snapmirror status' or the snapmirror log.

prodfiler4> vol move status production_vol_adm
Source                Destination                     CO Attempts    CO Time     State
production_vol_adm    ataggr0                         3              60          cutover

prodfiler4> Wed Feb 26 16:18:07 SGT [prodfiler4:vol.move.cutoverEnd:info]: Cutover finished for vol move of volume production_vol_adm to aggregate ataggr0 - time taken 14 secs

prodfiler4> vol move status production_vol_adm
Source                Destination                     CO Attempts    CO Time     State
production_vol_adm    ataggr0                         3              60          cutover

prodfiler4> Wed Feb 26 16:18:16 SGT [prodfiler4:wafl.vvol.renamed:info]: Volume 'ndm_dstvol_1393402490' renamed to 'production_vol_adm_old_1393402490'.
'ndm_dstvol_1393402490' renamed to 'production_vol_adm_old_1393402490'
Wed Feb 26 16:18:17 SGT [prodfiler4:vol.move.End:info]: Successfully completed move of volume production_vol_adm to aggr ataggr0.



I have observed one important thing during the DataMotion is that you SOURCE VOLUME should have atleast 10% of FREE_SPACE left in it if not you may face issue during cut_over time as once after the base transfer it creates a snapshot on the Source Volume and do the updates based on the snapshot if you don't have enough space to create snapshots it will just show you transferring updates for more than 5 times which means you have to abort the DataMotion and increase the SOURCE VOLUME size and then start over again.

One more Important thing is the snap autodelete commitment settings on the volume should be set to Try 

Ex:- snap autodelete production_vol_adm commitment try ( Set this before initiating DataMotion )

"Netapp" How to Convert a 32-Bit Aggregate in to a 64-Bit

Starting from Ontap 8.1 if you want to convert an aggregate of 32-bit to 64-bit all you need is add disks to grow the aggregate over 16TB and it will automatically convert in to 64-bit and it happens in the background (no downtime required ) from 32 to 64 bit.  Growing over 16TB is the safest method.

Example :- 

For example if you have 14 disks in a 32-Bit aggregate and then you may need to add additional 2 more disks to make it 64 Bit

Syntax:- 

Nayab >  aggr add aggr_name -64bit -upgrade normal -n 2

This will add 2 disks to the aggregate at the same time it will convert your aggregate to 64-Bit 


If you are not able to add disks then there is another alternative can do it through the Diag Mode ( But it is very risky to perform unless directed by a Netapp Personal )

It is your own risk If you want to try this method :-

Nayab > priv set diag 

Nayab *> aggr 64bit-upgrade start aggr1 -mode grow-all