Archive | App Volumes RSS feed for this section

Unable to uninstall/upgrade VMware Horizon Client within the VMware App Volumes AppStack

22 Jul

We had a very long ongoing issue wherein we couldn’t uninstall or upgrade the VMware Horizon Client within the AppStack. We had successfully installed the Horizon Client within the AppStack. However, when it was time to perform an upgrade or uninstall to the latest version, it would fail during a reboot with the following error.

Unknown HardError

We initially saw the issue on App Volumes 2.14. While we were troubleshooting for an extended period, we upgrade to App Volumes 2.18.1, and both the versions exhibited the same failure during uninstall or upgrade.

Process to reproduce the error

  • Upgrade horizon client –> reboot –> hard error
  • Uninstall horizon client –> reboot –>hard error
  • Uninstall horizon client –> install horizon client –> reboot –> hard error
  • Upgrade horizon client –> complete provisioning without reboot –> completes successful –> during next update of AppStack it crashes with Hard error
  • Uninstall horizon client –> complete provisioning without reboot –> completes successful –> during next update of AppStack it crashes with Hard error

Environment Details

VMware Horizon 7.11
VMware App Volumes 2.18.1
VMware Dynamic Environment Manager 9.10
VMware Horizon Client 5.x

Process of elimination

  • Upgrade the Horizon Client to the various 5.x version to remove any version specific Client related issues
  • We didn’t have Antivirus running on the AppStack capturing template
  • We could build the AppStack from scratch with the newer version of Horizon Client but only upgrade/uninstall would fail
  • We were honestly running out of troubleshooting ideas

Resolution

After trying out all the usual steps and avoid re-creating AppStack every single time during life cycle management, we managed to open a VMware GSS case handled by Karan Ahuja(Very helpful support engineer), which ended been worked by the engineering team(Art Rothstein – Champ in AV Eng Team). Note quite alot of logs and Procmon were exchanged from the problematic application capturing VM template. Finally, the fix was determined as a AppStack snapvol.cfg exclusion. After putting this exclusion into the AppStack – App capturing VM during provisioning we could upgrade or uninstall Horizon Client.

exclude_registry=\REGISTRY\MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\ProfileService
Path exclusion in AppStack snapvol.cfg

Disclaimer – Due to the nature of the issue and time taken to resolve it we decided to move the Horizon Client from AppStack into the base image. However, the fix is validated, and 100% working post the exclusion.

I hope you will find this information useful if you encounter the issue. A big thanks to Manivannan Arul my teammate for his continuous effort while troubleshooting with GSS over a period of 4+ months.

Thanks,
Aresh Sarkari

Black screen when re-connect to VMware Horizon virtual desktop

27 May

We had an issue after we upgraded our EUC Stack, especially VMware App Volumes 2.14 to 2.18.1. Quite a few end-users started reporting black screen when they were trying to re-connect to their desktops post the original session launch. This would mean re-connect post breaks, endpoint screen locks, next working day re-connections, etc.

EUC Environment Details:
VMware Horizon 7.11
VMware App Volumes 2.18.1
VMware Dynamic Environment Manager 9.10
VMware Horizon Client 5.x
VMware Workspace One 3.3

Process of elimination

  • If we re-created the writable volumes of the problematic end-users the black screen issue would go away. This provided us with a clue that the problem lied with VMware App Volumes – Writable Volumes
  • No errors/failures observed within the VMware DEM/Horizon logs
  • Upgrade the Horizon Client to the latest 5.x version to remove any Client related issues
  • We already had the necessary anti-virus exclusion based on VMware Antivirus Considerations in a VMware Horizon 7 Environment

Resolution
After trying out all the usual steps and avoid re-creating writable volumes for problematic end-users, we managed to open a VMware GSS case handled by Karan Ahuja(Very helpful support engineer), which ended been worked by the engineering team(Art Rothstein – Champ in AV Eng Team). Note quite alot of logs, memory dumps, and Procmon were exchanged from the problematic VM using various remote gathering techniques. Finally, the fix was determined as a writable volume snapvol.cfg exclusion. (In our case, the problem is caused by smss.exe using a copy of winlogon.exe that is on the writable volume). After putting this exclusion into all problematic end-users, they stopped seeing Black screen issues upon re-connect.

exclude_path=%SystemRoot%\System32\winlogon.exe
Path exclusion in writable volumes snapvol.cfg

In this blog, I am not outlining the steps on how to add the snapvol.cfg exclusion as my ex-colleague Daniel Bakshi outlines on a VMware blog post on how to do it step by step. I hope you will find this information useful if you encounter intermittent black screen issues.

Thanks,
Aresh Sarkari

Report all VMware App Volumes Writable Volumes with Status Disabled and Orphaned

22 Apr

Often within the App Volumes Manager, there are Writable Volumes that will show up as Status “Orphaned” and essentially that can be caused by active directory user accounts that have been disabled in AD.

Writable Status = Orphaned

There is also a Status called “Disabled” and that can be caused when an App Volumes administrator decides to disable the Writable Volumes.

Writable Status = Disabled

Now if you have a enteprise environment with 1000’s of users, it’s hard to perform this activity from the UI. I have created a script that can report on the status of “Orphaned” and “Disabled” send you the output in *.csv report on a daily/weekly basis as per your needs.

####################################################################
# Get List of Writable Volumes from AppVolumes Manager for Status Disabled and Orphaned
# Author - Aresh Sarkari (@askaresh)
# Version - V2.0
####################################################################

# Run at the start of each script to import the credentials
$Credentials = IMPORT-CLIXML "C:\Scripts\Secure-Creds\SCred_avmgr.xml"
$RESTAPIUser = $Credentials.UserName
$RESTAPIPassword = $Credentials.GetNetworkCredential().Password


$body = @{
    username = “$RESTAPIUser"
    password = “$RESTAPIPassword”
}

Invoke-RestMethod -SessionVariable DaLogin -Method Post -Uri "https://avolmanager.askaresh.com/cv_api/sessions” -Body $body

$output = Invoke-RestMethod -WebSession $DaLogin -Method Get -Uri "https://avolmanager.askaresh.com/cv_api/writables" -ContentType "application/json"

$output.datastores.writable_volumes | Select-Object owner_name, owner_upn, title, status | Where-Object {[string]$_.status -match "Orphaned" -and $_.title -match "(disabled)"} | Export-Csv -NoTypeInformation -Append D:\Aresh\Orphaned.Disabled-Writables.$(Get-Date -Format "yyyyMMddHHmm").csv

#send an email (provided the smtp server is reachable from where ever you are running this script)
$emailfrom = 'writablevolumes@askaresh.com'
$emailto = 'email1@askaresh.com', 'email2@askaresh.com'
$emailsub = 'Wrtiable Volumes with status Orphaned and Disabled - Weekly'
$emailbody = 'Attached CSV File from App Volumes Manager. The attachment included the API response for all the Writable which are orphaned and Disabled in UI'
$emailattach = "D:\Aresh\Orphaned.Disabled-Writables.$(Get-Date -Format "yyyyMMddHHmm").csv"
$emailsmtp = 'smtp.askaresh.com'

Send-MailMessage -From $emailfrom -To $emailto -Subject $emailsub -Body $emailbody -Attachments $emailattach -Priority High -DeliveryNotificationOption OnFailure -SmtpServer $emailsmtp

GitHub – https://github.com/askaresh/scripts/blob/master/wrtiable-orph-disa

Depending upon the output, you can have your service desk get in touch with the Active Directory teams to get the affected end-users to be removed from the App volumes writable volumes entitled groups and then proceed towards clean up of their writable volumes if there is no legal hold requirements.

I hope you will find this script useful to get a report for all writable volumes with status Orphaned and Disabled. My request if you further enhance the script or make it more creative, I hope you can share it back with me?

Thanks,
Aresh Sarkari

Report all VMware App Volumes Writable Volumes with low disk space

20 Apr

We have provided end-users with 30 GB Writable Volumes, and within the App Volumes Manager console there is an ability in the UI to see the Writable Volumes disk free under the view called – “Usage View”

Writable Volumes - Usage View
Writable Volumes – Usage View

The biggest challenge is if you have 1000’s of users, it’s hard to perform this activity from the UI. I have created a script that can send you the output in *.csv report on a daily/weekly basis as per your needs.

####################################################################
# Get List of Wrtiable Volumes from AppVolumes Manager for free space less than 3 GB out of 30 GB
# Author - Aresh Sarkari (@askaresh)
# Version - V2.0
####################################################################


# Run at the start of each script to import the credentials
$Credentials = IMPORT-CLIXML "C:\Scripts\Secure-Creds\SCred_avmgr.xml"
$RESTAPIUser = $Credentials.UserName
$RESTAPIPassword = $Credentials.GetNetworkCredential().Password


$body = @{
    username = “$RESTAPIUser"
    password = “$RESTAPIPassword”
}

Invoke-RestMethod -SessionVariable DaLogin -Method Post -Uri "https://avolmanager.askaresh.com/cv_api/sessions” -Body $body

$output = Invoke-RestMethod -WebSession $DaLogin -Method Get -Uri "https://avolmanager.askaresh.com/cv_api/writables" -ContentType "application/json"

$output.datastores.writable_volumes | Select-Object owner_name, owner_upn,total_mb, free_mb, percent_available, status | Where-Object {$_.free_mb -lt 3072}  | Export-Csv -NoTypeInformation -Append D:\Aresh\Writableslt3gb.$(Get-Date -Format "yyyyMMddHHmm").csv

#send an email (provided the smtp server is reachable from where ever you are running this script)
$emailfrom = 'writablevolumes@askaresh.com'
$emailto = 'email1@askaresh.com', 'email2@askaresh.com' #Enter your SMTP Details
$emailsub = 'Wrtiable Volumes Size (free_mb) less than 3 GB out of 30 GB - 24 Hours'
$emailbody = 'Attached CSV File from App Volumes Manager. The attachment included the API response for all the Writable Volumes less than 3 GB of free space'
$emailattach = "D:\Aresh\Writableslt3gb.$(Get-Date -Format "yyyyMMddHHmm").csv"
$emailsmtp = 'smtp.askaresh.com'

Send-MailMessage -From $emailfrom -To $emailto -Subject $emailsub -Body $emailbody -Attachments $emailattach -Priority High -DeliveryNotificationOption OnFailure -SmtpServer $emailsmtp

GitHub https://github.com/askaresh/scripts/blob/master/writablevolumesdiskusage

Depending upon the output, you can have your service desk get in touch with the affected end-users to clear-up disk space or provide options for further expansion.

I hope you will find this script useful to get a report for all writable volumes nearing their disk space usage. My request if you further enhance the script or make it more creative, I hope you can share it back with me?

Thanks,
Aresh Sarkari

Intermittent Clipboard issues on VMware Horizon virtual desktop

18 Apr

Recently, we had an issue within our environment where-in end-users complained of intermittently one-way clipboard not working(Virtual Desktop to Endpoint will fail). The tricky part here was it would happen intermittently to anyone without any set pattern.

Environment Details:
VMware Horizon 7.11
VMware App Volumes 2.18.1
VMware Dynamic Environment Manager 9.10
VMware Horizon Client 5.x

Process of elimination

  • We were not using the Horizon Blast GPO for setting the clipboard.
  • The clipboard was setup using DEM Horizon Smart Policies – Enabled Both Directions
  • Upgrade the Horizon Client to the latest version to remove any Client related issues
  • We already had the anti-virus process exclusion of VMwareViewClipboard.exe
  • We disabled the Writable Volumes, and the clipboard issue will never occur.

Resolution

The above test made it evident that something within the Writable Volumes was causing the intermittent clipboard issue. The next thing that came to mind is adding path/process exclusion within the snapvol.cfg. One may ask how did you determine that path, but recently we have had many application issues that needed exclusion to make them work.

What I didn’t know was which path or process, until the task manager showed a clipboard process for Horizon called – VMwareViewClipboard.exe and its Path – C:\Program Files\Common Files\VMware\Remote Experience\x64. I read many communities post having mentioned this process. However, I wasn’t sure if adding the entire path exclusion made sense as I could see many Horizon process *.exe and wasn’t sure what additional repercussions it can have. I went ahead, adding the below process exclusion.

exclude_process_name=VMwareViewClipboard.exe
Process exclusion in writable volumes snapvol.cfg

Post adding the exclusion, all the end-users with intermittent clipboard issues were always able to do two side clipboard. In this blog, I am not outlining the steps on how to add the snapvol.cfg exclusion as my ex-colleague Daniel Bakshi outlines on a VMware blog post on how to do it step by step.

Update 2nd May 2020
We had a VMware GSS support case open on the same issue, and they came back with a suggestion to exclude this registry path instead of the process exclusions. Note we been told there is no impact with process or registry, but its a good practice to do registry/path exclusions instead of the process. This registry/subkeys are responsible for the Clipboard – DEM Horizon Smart Policies.

exclude_registry=\REGISTRY\MACHINE\SOFTWARE\VMware, Inc.\VMware UEM
Process exclusion in writable volumes snapvol.cfg

I hope you will find this information useful if you encounter intermittent clipboard issues.

Thanks,
Aresh Sarkari

VMware EUC – Horizon, UAG, VIDM and AppVolumes – NSX Load Balancing – Health Check Monitors

2 Feb

There is no single place to find a consolidated list of Load balancer health check monitors (aka Service Monitors in NSX) for the VMware EUC products:

I have been using VMware NSX load balancer across the board. The below details will provide an overview of what to enter for the health monitors. Note – If you are using something more meaningful  for your environment leave feedback in the comments section. I will try to implement the same and update the blog later.

VMware Unified Access Gateway (UAG)

Create a new Service Monitor under NSX and call is UAG_https_monitor. Refer to the screenshot for more details.

UAG Service Monitor

Send String: GET /favicon.ico
Send String: GET /favicon.icoSend String: GET /favicon.ico
Send String: GET /favicon.ico
Send String: GET /favicon.ico

VMware Identity Manager (VIDM)

Create a new Service Monitor under NSX and call is VIDM_https_monitor. Refer to the screenshot for more details.

VIDM Service Monitor
Send String: GET /SAAS/auth/login
Response code: 200s

Horizon Connection Servers

Create a new Service Monitor under NSX and call is Horizon_https_monitor. Refer to the screenshot for more details.

image
Send String: GET /broker/xml/
Receive string: /styles/clientlaunch-default

VMware App Volumes

Create a new Service Monitor under NSX and call is AV_https_monitor. Refer to the screenshot for more details.

AV Service Monitor

I hope you will find these monitors useful in monitoring the VMware EUC products.

Thanks,
Aresh Sarkari


Export VMware App Volumes – Writable Volumes from VSAN Datastore

28 Nov

If you have a VMware VSAN environment and you wanted to export a App Volumes – Writable Volumes from the vsanDatastore to another datastore, storage or for VMware GSS/R&D for further analysis go ahead and read further!

Background – The traditional way of exporting the Writable Volumes from the source vsanDatastore was to attach the *.vmdk to a dummy VM as a “Existing Hard Disk” and export the dummy VM using the “Export OVF Template” option from the vCenter. Repeat all the steps on target datastore where-in it needs to be imported. However, if you want an alternate and easy method than the dummy VM then follow the below steps.

Step by Step Instructions:

— SSH to any ESXi Host Resource Cluster where the WV is stored and browser to the cloudvolumes/writable directory location:

# cd /vmfs/volumes/vsanDatastore/cloudvolumes/writable (This is the location where all end-user writable volumes are stored)

Now search for the end-user (E.g twood) for which you want to export the Writable Volumes.

# ls -lh | grep twood
User to export WV

Now lets open the *.vmdk file using “cat” command to retrieve the Object ID information. Make a note of the ObjectID

# cat DOMAIN!5C!twood.vmdk

Cat to find ObjectID

In my scenario the Object ID was properly pre-created I didn’t have to use the objtool to find out the Object opened. However, in some cases you might have to run the following command

# /usr/lib/vmware/osfs/bin/objtool open -u (Where Object ID is displayed using the ‘cat’ command

This screenshot below is an example of situations where in the Object ID is not properly pre-created. It will provide you with an output Object Opened at path:
Objecttool Output

Now using WINSCP login to the same ESXi Host and go to the path:
Object ID – /vmfs/device/vsan/d17efe58-5610-4dd2-0d9e-ecf4bbea2830 (my scenario)
Or/else Object opened at path in the screenshot above.

Download the file “d17efe58-5610-4dd2-0d9e-ecf4bbea2830” which is Writable Volumes (*.vmdk) file and move the files to local or remote location that you are using the WINSCP tool.

— Rename the Object ID to a friendly name shown in the cloudvolumes/writable Directory Folder. I renamed it (DOMAIN!5C!twood.vmdk)

You don’t need the *.vmdk.metedata file

The Writable Volumes is now exported out of the VSAN environment you can attach the *.vmdk to a non App Volumes Agent machine to look at the contents inside the Writable Volumes. If you are going to send the vmdk to VMware GSS/R&D make sure to zip it before uploading

I hope you will find these steps useful and help you export a Writable Volume from your vsanDatastore. I haven’t been able to try AppStacks with this method its on my to-do list.

Thanks,
Aresh Sarkari

McAfee Exclusion for VMware App Volumes 2.x – 100% CPU Issues

27 Nov

In your Virtual Desktop Infrastructure with the following configurations:

If you start noticing 100 % – CPU Usage for prolonged period of time and the Horizon Session getting disconnected from time to time after launch then you might need to include the following exclusion within your Writable Volumes (UIA+Profile) snapvol.cfg file:

#McAfeeExclusion
exclude_process_path=\Program Files\Common Files\McAfee\SystemCore

My colleague Daniel Bakshi has written an extensive blogpost on how to modify the snapvol.cfg for individual or group of end-users please reference it to make the necessary changes – Using the VMware App Volumes snapvol.cfg File to Customize Writable Volumes

I hope you will find these exclusion useful and will help you resolve a similar issue a lot quicker. A big thanks to Art Rothstein in helping to troubleshoot and resolve the issue.

Thanks,
Aresh Sarkari

Error 1303 The installer has insufficient privileges to access this directory – Upgrade from App Volumes 2.12 to 2.12.1

12 Apr

With the latest version of App Volumes 2.12.1, you don’t have to uninstall the older version of App Volumes Manager. The latest App Volumes Manager 2.12.1 installer takes care of uninstalling, fresh-install and retain all the configuration details and settings automatically for you.

During the upgrade I encountered the following error:

“Error 1303. The installer has insufficient privileges to access this directory: C:\Program Files(x86)\CloudVolumes\Manager\log. The installation cannot continue. Log on as an administrator or contact your system administrator.”

App Volumes Upgrade Error

Resolution:
In our scenario we have VMware vRealize Log Insight Agent installed on the App Volumes Manager VM’s which is doing Syslog. The Log Insight agent captures the logs(production.log) inside the folder “C:\Program Files(x86)\CloudVolumes\Manager\log”. As the service is in the running state, it didn’t allow the folder to delete and left a ghost folder on the filesystem.

Log Insight Agent Service

After going into the services.msc and stopping the VMware vRealize Log Insight Agent service and click Retry, the setup manages to complete the upgrade successfully.

I hope this workaround helps you during your upgrade if you encounter a similar error message.

Thanks,
Aresh

Export Writable Volumes from vSAN Datastore

15 Nov
In certain scenarios such as uploading the Writable Volumes *.vmdk to VMware support team to analyze issues due to Writable Volumes or you simply want to export the WV from one vSAN datastore to another vCenter or vSAN Datastore
Following is the step by step procedure to export Writable Volumes from vsanDatastore for troubleshooting purposes:

Source vCenter or vSAN Datastore:

  • Create a dummy VM (No need to power on the VM)
  • Add a HDD to the dummy VM – Use existing disk option – Locate the Writable Volumes under –  /vmfs/volumes/vsandatastore/cloudvolumes/writables) and click OK
  • Now you can export the dummy VM as a OVA or OVF to another vCenter or vSAN datastore
  • Save the OVA to a File Share or GSS FPT for further troubleshooting

Target vCenter or vSAN Datastore

  • Import the OVA into the target vCenter
  • SSH to a host in the cluster from which the Writable Volumes (WV vmdk) needs to be copied to the correct path cd /vmfs/volumes/vsandatastore/cloudvolumes/writables
  • Copy the files *.vmdk from dummy VM Folder to the writable folder
    • cp /vmfs/volumes/DummyVM/AV-WV/domainname!5C.aresh.vmdk /vmfs/volumes/vsandatastore/cloudvolumes/writable
  • Go to App Volumes Manager – Writable Volumes – Import Writable Volumes
  • Now you should see the writable for that user
Following are the step the engineer needs to perform for further troubleshooting it can be GSS, R&D or L3.
  • Import the template into the environment
  • Click on convert to virtual machine
  • On any existing Windows 7 VM without AV Agent (make sure not AV agent is installed). One needs to have a Windows 7 VM pre-build
  • Add HDD and select the existing disk option. Search for the vmdk in the folder previously imported
  • Assign the volume a driver letter and you can browse the contents of the WV
  • Troubleshoot further!

I hope this post will save you a lot of time when exporting WV from VSAN Datastore

Thanks,
Aresh