Netapp Not able to access CIFS shares

 We May not be able to access CIFS shares ( including C$ and ETC$ ) on a New Filer I have come across this issue and i tried all possible ways to figure out the cause and finally i was able to fix the issue, 

The reason is when you run CIFS SETUP on your filer it will create a user called "PCUSER"  ( All windows users will be mapped to this users while accessing the shares on the filer )  , you can check it from the below command your file should be similar to below

FAS3220> rdfile  /etc/passwd

root : : 0 : : : / :
pcuser : : 65534 : 65534 : : / :
nobody : : 65534 : 65534 : : / :
ftp : : 65533 : 65533 : FTP Anonymous : /home/ftp :


For some reason  it does not create the user pcuser and i have noticed the /etc/passwd was empty so all what i did is to copy the /etc/passwd from one of my old filers and don't forget to run source command after copying

FAS3220> source  /etc/passwd

Once done you should be able to access you shares

Happy Knowledge Sharing :) 

Migrating a root Volume in Netapp

  Please remember that moving a root volume requires a reboot of the controller.  
Therefore, i suggest to complete this activity in your Maintenance windows. 

It is always good to have a snapshot created before starting any migrations 


FASProd>snap create -V vol0 vol0_snap


 1.       Disable the cluster:  

FASProd>cf disable

2.       Check the size of current vol0:  

FASProd>df -Vh  vol0

3.       Create a new root volume on destination aggr:    

FASProd>vol create  vol0_new  dest_aggr  <SIZE>

4.       Copy the data to the new volume:

FASProd> ndmpcopy /vol/vol0  /vol/vol0_new ( Can also use Vol Copy )

 5.       Once copy is done now Rename the old root volume: 

FASProd>vol rename vol0 vol0_old

6.       Rename the new root volume:  

FASProd>vol rename vol0_new vol0

7.       Now change the new vol0 to be used as root volume 

FASProd> vol options vol0 root

8.       Reboot the controller:  

FASProd>reboot

9.       Confirm the destination aggr now hosts the root vol0:  

FASProd>vol status vol0  

10.   Once confirmed proceed to Offline and destroy the old volume

FASProd> vol offline vol0_old  

FASProd> vol destroy vol0_old

Now i am a NTSP ( Nimble Technical Sales Professional )

Knowing what is Vmware Vcenter, ESXi, Vsphere

I was really confused with the terms Vsphere , Vcenter and ESXi and i have thought to write up the difference so that it might help someone at some point of time

What is vSphere ?

vSphere is suite of products packaged shipped by VMware virtualization company, which includes products. Like VMware ESXi hypervisor, VMware vCenter Server, etc.

VMware ESXi ?

ESXi or hypervisor is bare metal OS which virtualizes x86 server hardware. You use ESXi ( OS ) installation  media to install OS on top of hardware just like you install any other OS like Windows or Linux

Once you have VMware ESXi installed on a x86 server hardware I is ready to virtualized. What does it mean ? - It means you can now use same server to create , configure, run more than one virtual machines and install required Operating systems within Virtual Machines they are referred as Guest Operating systems which could be Windows or Linux etc

 What is vCenter ?

 

Vcenter is another Vmware product used to manage one or more ESXi servers comes in the Vsphere software package.

Netbackup Vmoprcmd Cobra Communication Error

If you have your Netbackup Master Server Version 7.6 and your media server ( We had 7.0.1 ) version lower than master version you may face the error while you run the command Vmoprcmd 

 Even the compatibility matrix says 7.6 is compatible with 7.0.1 but still we have faced the below error as soon as we upgraded the master server from 7.0.1 to 7.6.0.1,

Cobra Communication Error 

Solution :-  Is to Upgrade Media server also from 7.0.1 to 7.6.0.1 to fix this issue.

Netbackup Oracle backup Error 6 On HP-UX Client

One of our Oracle DB backups were failing with error 6 which is a generic error for DB backups using Netbackup and from the error log i have found

Errors in file /OraBase/admin/Test50/bdump/Test50_j000_29761.trc:

ORA-12012: error on auto execute of job 661138

ORA-00257: archiver error. Connect internal only, until freed.




Error:- ORA-00257: archiver error. Connect internal only, until freed.

Solution:- 

And i see this is moreover Oracle Side to troubleshoot rather the Netbackup side and after a small research i have found the solution is to move some of the archive logs 

--> find the location of Archive destination by
 show parameter archive_dest
 
 lets say it provide  LOCATION=/Test50/oradata/mydb/arch

-->  move some files to some other location using os command
 cd /Test50/oradata/mydb/arch
 mv /Test50/oradata/mydb/arch/* /Test5/oradata/mydb/arch-bkp/
 
If you want to do it from RMAN , try as below.

rman target /

RMAN> backup archive log all format '/Test50/oradata/mydb/arch-bkp';

RMAN> delete archive until time 'trunc(sysdate)';

Error:- ORA-12012: error on auto execute of job 7863452

Solution:- 

The underlying table required as part of one of the schedule maintenance tasks has invalid values. 
The related table is Test50.Testing_Base. 

After taking a look at the job log we see the failed execution attempts:

SQL> select log_date,status from dba_scheduler_job_run_details 
     where job_name='Test50_DBTesting_JOB';

LOG_DATE                              STATUS
------------------------------------- ------------------------------
11-FEB-14 10.00.07.3668 AM -06:00    FAILED
04-FEB-14 08.00.05.59665 AM -06:00   FAILED


2 rows selected.

Solution

Check the invalid values on the table and remove them.

SQL> select * from Test50.Testing_Base;

      DBID INSTANCE_NAME    BASELINE_ID BSLN_GUID                        TI A STATUS           LAST_COMPUT
---------- ---------------- ----------- -------------------------------- -- - ---------------- -----------
4056791929  orcl                       0 24T6YH78JNSHDT89ONSNS09Q2NND8728 NW N ACTIVE           140917:2200
4052355478  ORCL3                      0 12MDNSIdGAJKW474IETJBESGJH539930 ND Y ACTIVE           141022:2000


As we can see the above orcl is corrupted , Now we proceed to delete it.


The corrupt row can be removed as below.


SQL> DELETE FROM Test50.Testing_Base WHERE INSTANCE_NAME ='orcl';

1 row deleted.

SQL> commit;

Commit complete.

Manually re-execute the job and check the execution log, it must show the job executed successfully.  It takes a couple of minutes after execution to show the results in the log table.

SQL> exec dbms_scheduler.run_job('Test50.Testing_Base',false);
PL/SQL procedure successfully completed.


The issue was fixed, validate it by querying

SQL> select log_date,status from dba_scheduler_job_run_details 
     where job_name='Test50.Testing_Base';

LOG_DATE                              STATUS
------------------------------------- ------------------------------
14-FEB-14 11.00.07.315077 AM -06:30   FAILED
15-FEB-14 11.00.05.595559 AM -06:30   FAILED
17-FEB-14 03.41.20.714453 Am -06:30   SUCCEEDED

Now try re-run the job it should be successful :)

Netapp Volume DataMotion Highlights

Today i have performed Netapp DataMotion for volumes to migrate volumes from One Aggregate to another aggregate and i can see DataMotion uses SNAPMIRROR in the back end to move the data from OLD VOLUME to the NEW VOLUME. Please find the brief summary below how it works...

Note :- My volumes has ORACLE DB running on them and i have seen no disruption to the application during DataMotion it went smooth and Hassle Free.

prodfiler4> vol move start production_vol_adm ataggr0 -k

Use " -k " option if you want to keep the source volume as once after the VOL MOVE , Netapp destroys the source volume as all the new reads and writes were directed to the New Volume ( Destination Volume)

prodfiler4> Wed Feb 26 16:14:50 SGT [prodfiler4:vol.move.Start:info]: Move of volume production_vol_adm to aggr ataggr0 started
Creation of volume 'ndm_dstvol_1393402490' with size 21474836480  on containing aggregate
'ataggr0' has completed.

Volume 'ndm_dstvol_1393402490' is now restricted.

Wed Feb 26 16:15:13 SGT [prodfiler4:vol.move.transferStart:info]: Baseline transfer from volume production_vol_adm to ndm_dstvol_1393402490 started.
Transfer started.

Monitor progress with 'snapmirror status' or the snapmirror log.

Wed Feb 26 16:16:54 SGT prodfiler4:vol.move.transferStatus:info]: Baseline transfer from volume production_vol_adm to ndm_dstvol_1393402490 took 97 secs and transferred 1935528 KB data.

Wed Feb 26 16:16:56 SGT [prodfiler4:vol.move.transferStart:info]: Update from volume production_vol_adm to ndm_dstvol_1393402490 started.
Transfer started.

Monitor progress with 'snapmirror status' or the snapmirror log.

Wed Feb 26 16:17:18 SGT [prodfiler4:vol.move.transferStatus:info]: Update from volume production_vol_adm to ndm_dstvol_1393402490 took 13 secs and transferred 1160 KB data.

Wed Feb 26 16:17:23 SGT [prodfiler4:vol.move.transferStart:info]: Update from volume production_vol_adm to ndm_dstvol_1393402490 started.
Transfer started.

Monitor progress with 'snapmirror status' or the snapmirror log.

Wed Feb 26 16:17:44 SGT [prodfiler4:vol.move.transferStatus:info]: Update from volume production_vol_adm to ndm_dstvol_1393402490 took 12 secs and transferred 1104 KB data.

Wed Feb 26 16:17:44 SGT [prodfiler4:vol.move.updateTimePrediction:info]: Expected time for next update from volume production_vol_adm to ndm_dstvol_1393402490 is 12 secs to transfer 272 KB data.

Wed Feb 26 16:17:52 SGT [prodfiler4:vol.move.cutoverStart:info]: Cutover started for vol move of volume production_vol_adm to aggr ataggr0.
Transfer started.

Monitor progress with 'snapmirror status' or the snapmirror log.

prodfiler4> vol move status production_vol_adm
Source                Destination                     CO Attempts    CO Time     State
production_vol_adm    ataggr0                         3              60          cutover

prodfiler4> Wed Feb 26 16:18:07 SGT [prodfiler4:vol.move.cutoverEnd:info]: Cutover finished for vol move of volume production_vol_adm to aggregate ataggr0 - time taken 14 secs

prodfiler4> vol move status production_vol_adm
Source                Destination                     CO Attempts    CO Time     State
production_vol_adm    ataggr0                         3              60          cutover

prodfiler4> Wed Feb 26 16:18:16 SGT [prodfiler4:wafl.vvol.renamed:info]: Volume 'ndm_dstvol_1393402490' renamed to 'production_vol_adm_old_1393402490'.
'ndm_dstvol_1393402490' renamed to 'production_vol_adm_old_1393402490'
Wed Feb 26 16:18:17 SGT [prodfiler4:vol.move.End:info]: Successfully completed move of volume production_vol_adm to aggr ataggr0.



I have observed one important thing during the DataMotion is that you SOURCE VOLUME should have atleast 10% of FREE_SPACE left in it if not you may face issue during cut_over time as once after the base transfer it creates a snapshot on the Source Volume and do the updates based on the snapshot if you don't have enough space to create snapshots it will just show you transferring updates for more than 5 times which means you have to abort the DataMotion and increase the SOURCE VOLUME size and then start over again.

One more Important thing is the snap autodelete commitment settings on the volume should be set to Try 

Ex:- snap autodelete production_vol_adm commitment try ( Set this before initiating DataMotion )

"Netapp" How to Convert a 32-Bit Aggregate in to a 64-Bit

Starting from Ontap 8.1 if you want to convert an aggregate of 32-bit to 64-bit all you need is add disks to grow the aggregate over 16TB and it will automatically convert in to 64-bit and it happens in the background (no downtime required ) from 32 to 64 bit.  Growing over 16TB is the safest method.

Example :- 

For example if you have 14 disks in a 32-Bit aggregate and then you may need to add additional 2 more disks to make it 64 Bit

Syntax:- 

Nayab >  aggr add aggr_name -64bit -upgrade normal -n 2

This will add 2 disks to the aggregate at the same time it will convert your aggregate to 64-Bit 


If you are not able to add disks then there is another alternative can do it through the Diag Mode ( But it is very risky to perform unless directed by a Netapp Personal )

It is your own risk If you want to try this method :-

Nayab > priv set diag 

Nayab *> aggr 64bit-upgrade start aggr1 -mode grow-all

How to change DB instance job schedule for Netapp Snapmanager for SQL Server Busy message

Hi All We have been facing issue with one of our SQL DB instance which is backed up through SNAPMANAGER has been failing with status Snapmanager Server Busy.... So what i have thought is if i change the schedule where no other instances were getting backed up ..... It is always good to have atleast 30mins-1hr between each instance. Please take note that SnapManager for SQL will not follow the windows  task scheduler ( Snapmanager for Exchnage uses windows Task Scheduler ). All i have done to avoid the server busy message is to change the schedule as below and it worked perfectly. Now my backups are successful.

Open SQL Server  Management Studio

Select the DB Instance for which you want to change the SCHEDULE

Now go to SQL Server Agent àDB_Schedule_Nameà Properties

 Select the Job and click Edit

Now you will be able to check and change the scheduled time to run the snapmanager for SQL

Netbackup Error 830,96,252........ Due to Tape Library IBM 3500 Gripper Failure

We have our backups failing with errors 830, 96, 252, etc... Note that we are using Symantec Netbackup. I was not able to find the cause first thinking it was drives errors as our library was not serviced since years. Later my lead suspected beyond the drive errors then he found the hardware issue.

Gripper probably has a tape stuck in it. I have opened the door and turn the robot towards me so i can see if there are tapes inside and i have discovered a tape was stuck inside the gripper then there's a small mechanism on it, like a plastic rope that you can move with your finger. Pushing it back allows you to get the tape from the gripper. As after removing the tape from the gripper still it didn’t solve my issue then later the IBM engineer came down to our site and replaced the two grippers as they both went faulty. Now my backups are just running fine.

You can find the faulty Gripper in the below pics.