Archive | vSAN RSS feed for this section

Update VMware vSAN Storage Controller Firmware and Driver – In three easy steps

9 Jul

We were having frequent hardware issues on our Dell PowerEdge R630 hyper converged servers and the Dell Support recommended in upgrading the Avago (LSI) Dell PERC H730 Mini Controller Driver and Firmware to the latest version.

Avago (LSI) Dell PERC H730 Mini

Existing Version

Dell Recommended

Firmware

25.5.2.0001

25.5.4.0006

Driver

6.910.18.00

7.703.18.00-1OEM

Note: The procedure of upgrade is applicable and tested on vSphere 6.5 U1 or vSAN 6.6 environments

We were running the support Driver/Firmware as all vSAN Health checks were green. However, there was a latest build available which included few additional fixes. Let me show you how easy it was to update the Storage Controller Drivers and Firmware using the VMware Update Manager and vSAN Configuration Assistant Upgrade Tool.

The Sequence that we need to execute is as follows:

  • Upgrade the Driver – LSI_mr3 VIB driver using the VMware Update Manager
  • After doing the above step you will start seeing the Controller Firmware and Avago Management Tool listed in the vSAN Configuration Assistant Update Tool

Step 1 – Sync Cluster to Perform Online Test

Add the my.vmware.com account in the vCenter to enable checking/sync with the online engine

  • Enter the Username and Password
vSAN Build Recommendation Engine

Step 2 – Update Storage Controller Driver

The VMware Update Manager (VUM) will perform the Controller Driver Update in rolling reboot fashion one ESXi Host at a time

  • Select the Cluster and Choose the lsi_mr3: Avago Native MegaRAID SAS driver
Update Manager - LSI Controller Driver Package
  • Click on the Remediate Button
    • Select VSAN Cluster under Baseline Groups and the VIB Driver LSI under Baselines
    • Click Next
VUM Baselines Selection
  • Select all the Host in the Cluster (E.g. If you want to perform a quick test you can select one-host). In our case we selected all the 21 hosts
VUM - Select the Host

  • Select the Package
VUM - Select the Package

  • Click on Ignore warning
VUM - Ignore the warning

  • Select Do Not Change Power State and leave the timings to defaults
VUM - Power State

  • Select the three options as below
    • Disable DPM
    • Disable HA admission control
    • Migrate Powered-off VM
VUM - Cluster Remediation Options

  • Click Finish
    • Click on Pre-check Remediation and see the current configuration
    • The upgrade will start in a rolling reboot (1-by-1)
VUM - Configuration Preview

  • Make sure to verify the versions of the driver is showing updated/passed in the VSAN health tests

Step 3 – Storage Controller Firmware

Once the Driver is installed the Controller Management Tools and Firmware for Avago get listed together in the vSAN Configuration Assistant Update tool.

Both the components get installed onto the ESXi host together:

  • Click on Download and that will turn the status in Ready to install
  • Click on the Update All  and select the second option Rolling reboot (one host at a time)
  • Install the Firmware on all the ESXi Host within the cluster.
vSAN - Configuration Assistant Update Tool

  • Make sure to verify the versions of the firmware is showing updated/passed in the VSAN health tests

Check the Up-time of the ESXi host (Check every 2 hours and update the tracker)

This step will enable you to track the progress of the cluster as on how many host are done. In our scenario for the Driver (8 hr) and Controller (8 hr) combined together for a 21 host cluster it took close to 16 hours. Off course this number will vary depending upon the cluster usage.

  • Connect to the vCenter Linked mode

Connect-VIServer -Server rack-1-vc-5.domain.com -Protocol https -AllLinked

  • Check the Uptime of ESXi Host

Get-VMHost | Get-View | select Name, @{N="Uptime"; E={(Get-Date) - $_.Summary.Runtime.BootTime}}

I hope you will find these steps useful in upgrading the VSAN Controller/Driver firmware easily using the Update Manager + Configuration Assist Update Tool. Let me know, if you have additional questions in the comments section.

Thanks,
Aresh Sarkari

Export VMware App Volumes – Writable Volumes from VSAN Datastore

28 Nov

If you have a VMware VSAN environment and you wanted to export a App Volumes – Writable Volumes from the vsanDatastore to another datastore, storage or for VMware GSS/R&D for further analysis go ahead and read further!

Background – The traditional way of exporting the Writable Volumes from the source vsanDatastore was to attach the *.vmdk to a dummy VM as a “Existing Hard Disk” and export the dummy VM using the “Export OVF Template” option from the vCenter. Repeat all the steps on target datastore where-in it needs to be imported. However, if you want an alternate and easy method than the dummy VM then follow the below steps.

Step by Step Instructions:

— SSH to any ESXi Host Resource Cluster where the WV is stored and browser to the cloudvolumes/writable directory location:

# cd /vmfs/volumes/vsanDatastore/cloudvolumes/writable (This is the location where all end-user writable volumes are stored)

— Now search for the end-user (E.g twood) for which you want to export the Writable Volumes.

# ls -lh | grep twood
User to export WV

— Now lets open the *.vmdk file using “cat” command to retrieve the Object ID information. Make a note of the ObjectID

# cat DOMAIN!5C!twood.vmdk

Cat to find ObjectID

— In my scenario the Object ID was properly pre-created I didn’t have to use the objtool to find out the Object opened. However, in some cases you might have to run the following command

# /usr/lib/vmware/osfs/bin/objtool open -u (Where Object ID is displayed using the ‘cat’ command

This screenshot below is an example of situations where in the Object ID is not properly pre-created. It will provide you with an output Object Opened at path:
Objecttool Output

— Now using WINSCP login to the same ESXi Host and go to the path:
Object ID – /vmfs/device/vsan/d17efe58-5610-4dd2-0d9e-ecf4bbea2830 (my scenario)
Or/else Object opened at path in the screenshot above.

Download the file “d17efe58-5610-4dd2-0d9e-ecf4bbea2830” which is Writable Volumes (*.vmdk) file and move the files to local or remote location that you are using the WINSCP tool.

— Rename the Object ID to a friendly name shown in the cloudvolumes/writable Directory Folder. I renamed it (DOMAIN!5C!twood.vmdk)

— You don’t need the *.vmdk.metedata file

— The Writable Volumes is now exported out of the VSAN environment you can attach the *.vmdk to a non App Volumes Agent machine to look at the contents inside the Writable Volumes. If you are going to send the vmdk to VMware GSS/R&D make sure to zip it before uploading

I hope you will find these steps useful and help you export a Writable Volume from your vsanDatastore. I haven’t been able to try AppStacks with this method its on my to-do list.

Thanks,
Aresh Sarkari

Create a Memory Dump from a Suspended Virtual Machine – VMware vSAN

10 Nov

If you have a VMware VSAN environment and you wanted to capture a memory dump of the Virtual Machine for debugging or want to provide memory.dmp to VMware GSS or R&D for further analysis go ahead and read further!

Use Case – In our scenario had a few VDI Desktops running Windows 10 1607 + Horizon 7.3.1 + App Volumes Writable Volumes 2.13.1 + UEM 9.2.1 that were getting into unresponsive state. As a last resort we wanted to capture the memory dump to find out more what is causing the VM to get unresponsive.

Step by Step Instructions:

— Using the vCenter console select the Virtual Machine VM – Power – Suspend

— This will create the *.vmss and *.vmem file for Debugging. (Note the *.vmem file is applicable for ESXi 6.0 onwards)
VM Directory

— Make a note of the ESXi host Name/IP for the VM is in Suspend state

— SSH to the ESXi Host and browser to the VM Directory location:

# cd /vmfs/volumes/vsanDatastore/od-av-troub-1 (Where “od-av-troub-1” is the VM name)


— Now lets open the *.vmem file using “cat” command to retrieve the Object ID information. Make a note of the ObjectID

# cat od-av-trou-1-7622414e.vmem

Object ID

— In my scenario the Object ID was properly pre-created I didn’t have to use the objtool to find out the Object opened. However, in some cases you might have to run the following command

— Now using WINSCP login to the same ESXi Host and go the path:
Object ID – /vmfs/device/vsan/2c86055a-573b-d20a-5cdf-ecf4bbea1e48 (my scenario)
Or/else Object opened at path and download the file “2c86055a-573b-d20a-5cdf-ecf4bbea1e48” which is your ”*.vmem file and move the files to local or remote location that you are using the WINSCP tool.

— Rename the Object ID to a friendly name shown in the VM Directory Folder. I renamed it (od-av-trou-1-7622414e.vmem)

— For the *.vmss (od-av-trou-1-7622414e.vms) you can directly WINSCP to the ESXi Host and go to the location in the table and move the files to your local or remote location

— Once you have both the files *.vmem and *.vmss you can use a VMware Vmss2core Fling and convert it to a dump. Please make sure you meet the requirements and use the appropriate switches to your environment

# vmss2core -W8 od-av-trou-1-7622414e.vmss od-av-trou-1-7622414e.vmem 

— The above command will generate a memory.dmp file which can used in WINDBG for further analysis. If you are sending the dump file to someone make sure use *.zip and compress it before sending.

I hope you will find these steps useful and save a lot of time during daunting unresponsive VM issues. A big thanks to Frank EscarosBuechsel to helping with the entire procedure.

Thanks,
Aresh Sarkari

Export Writable Volumes from vSAN Datastore

15 Nov
In certain scenarios such as uploading the Writable Volumes *.vmdk to VMware support team to analyze issues due to Writable Volumes or you simply want to export the WV from one vSAN datastore to another vCenter or vSAN Datastore
Following is the step by step procedure to export Writable Volumes from vsanDatastore for troubleshooting purposes:

Source vCenter or vSAN Datastore:

  • Create a dummy VM (No need to power on the VM)
  • Add a HDD to the dummy VM – Use existing disk option – Locate the Writable Volumes under –  /vmfs/volumes/vsandatastore/cloudvolumes/writables) and click OK
  • Now you can export the dummy VM as a OVA or OVF to another vCenter or vSAN datastore
  • Save the OVA to a File Share or GSS FPT for further troubleshooting

Target vCenter or vSAN Datastore

  • Import the OVA into the target vCenter
  • SSH to a host in the cluster from which the Writable Volumes (WV vmdk) needs to be copied to the correct path cd /vmfs/volumes/vsandatastore/cloudvolumes/writables
  • Copy the files *.vmdk from dummy VM Folder to the writable folder
    • cp /vmfs/volumes/DummyVM/AV-WV/domainname!5C.aresh.vmdk /vmfs/volumes/vsandatastore/cloudvolumes/writable
  • Go to App Volumes Manager – Writable Volumes – Import Writable Volumes
  • Now you should see the writable for that user
Following are the step the engineer needs to perform for further troubleshooting it can be GSS, R&D or L3.
  • Import the template into the environment
  • Click on convert to virtual machine
  • On any existing Windows 7 VM without AV Agent (make sure not AV agent is installed). One needs to have a Windows 7 VM pre-build
  • Add HDD and select the existing disk option. Search for the vmdk in the folder previously imported
  • Assign the volume a driver letter and you can browse the contents of the WV
  • Troubleshoot further!

I hope this post will save you a lot of time when exporting WV from VSAN Datastore

Thanks,
Aresh

Default VM Storage Policies (View Auto Created) for Horizon View 6 and VSAN

24 Sep


I love VSAN! Horizon View 6 and VSAN are like a marriage made in heaven, if you create Linked Clone or Full Clone desktops in View 6.0 using VSAN. It automatically adds the default Virtual Machine (VM) Storage Policies to the objects that are created during the desktop pool creation using View Administrator. The objects that get created using the desktop pool creation such as Persistent Disk, Replica Disk etc. automatically add the disk stripe, Flash cache and fault tolerance to each object.
What does that mean? Out of the box, your desktop objects are made highly available and one doesn’t have to go to individual desktops and set VM Storage policies. The following table outlines the View Auto Created out-of-the-box VM Storage Policies available with Horizon View 6 and VSAN:

Default VM Storage Policies
No. Disk stripes per object
Flash read cache reservation (%)
No. Failures to tolerate
Object space reservation (%)
Desktop Type
VM_HOME_
1
0
1
0
 Linked Clone/Full Clone
OS_DISK_
1
0
1
0
 Dedicated Linked Clone
REPLICA_DISK_
1
10
1
0
 Linked Clone
Persistent Disk_
1
0
1
100
 Linked Clone/Full Clone
FULL_CLONE_DISK_FLOATING_
1
0
0
100
 Floating Full Clone
FULL_CLONE_DISK_
1
0
1
100
 Dedicated Full Clone
OS_DISK_FLOATING_
1
0
0
0
 Floating Linked Clone

More Details:
VM_HOME – The folder which contains the virtual machine configuration files, such as .vmx, .vmsd, and .nvram.
OS_Disk – Linked Clone Dedicated/Persistent desktop pools
OS_DISK_FLOATING – Linked Clone Floating/Non-persistent desktop pools
REPLICA_DISK – A replica of the parent VM is created and stored on 1 per LUN basis from which the LC desktops are created
Persistent Disk – The User Persistent disk that redirects the user profile and user data to D: drive
_ – The GUID seen at the end of the default policy is a cluster GUID for a view deployment. The policies can be shared amongst multiple pools within a single view deployment.
Number of Failures to Tolerate (FTT) – Defines the number of hosts, disk, or network failures a storage object can tolerate. For n failures tolerated, n+1 copies of the object are created and 2n+1 hosts that contribute storage are required.
Number of Disk Stripes Per Object – The number of HDDs across which each replica of a virtual machine object is striped. A value higher than 1 might result in better performance, but also results in a higher use of system resources.
I hope this information helps. Feel free to post your comments down below.
Best Regards,
Aresh Sarkari