Black screen when re-connect to VMware Horizon virtual desktop

27 May

We had an issue after we upgraded our EUC Stack, especially VMware App Volumes 2.14 to 2.18.1. Quite a few end-users started reporting black screen when they were trying to re-connect to their desktops post the original session launch. This would mean re-connect post breaks, endpoint screen locks, next working day re-connections, etc.

EUC Environment Details:
VMware Horizon 7.11
VMware App Volumes 2.18.1
VMware Dynamic Environment Manager 9.10
VMware Horizon Client 5.x
VMware Workspace One 3.3

Process of elimination

  • If we re-created the writable volumes of the problematic end-users the black screen issue would go away. This provided us with a clue that the problem lied with VMware App Volumes – Writable Volumes
  • No errors/failures observed within the VMware DEM/Horizon logs
  • Upgrade the Horizon Client to the latest 5.x version to remove any Client related issues
  • We already had the necessary anti-virus exclusion based on VMware Antivirus Considerations in a VMware Horizon 7 Environment

Resolution
After trying out all the usual steps and avoid re-creating writable volumes for problematic end-users, we managed to open a VMware GSS case handled by Karan Ahuja(Very helpful support engineer), which ended been worked by the engineering team(Art Rothstein – Champ in AV Eng Team). Note quite alot of logs, memory dumps, and Procmon were exchanged from the problematic VM using various remote gathering techniques. Finally, the fix was determined as a writable volume snapvol.cfg exclusion. (In our case, the problem is caused by smss.exe using a copy of winlogon.exe that is on the writable volume). After putting this exclusion into all problematic end-users, they stopped seeing Black screen issues upon re-connect.

exclude_path=%SystemRoot%\System32\winlogon.exe
Path exclusion in writable volumes snapvol.cfg

In this blog, I am not outlining the steps on how to add the snapvol.cfg exclusion as my ex-colleague Daniel Bakshi outlines on a VMware blog post on how to do it step by step. I hope you will find this information useful if you encounter intermittent black screen issues.

Thanks,
Aresh Sarkari

Tags: , , , ,

Swagger-UI and Postman Collection for VMware Unified Access Gateway

6 May

I aimed to perform a particular VMware Unified Access Gateway (UAG) tasks programatically. After some guidance from Mark Benson he introduced me to the Swagger-UI that is available within the product.

To access the Swagger-UI on UAG open the following URL within the browser and enter your username and password.

https://uagnameorip:9443/swagger-ui/index.html
Swagger-UI – UAG API Calls

One can do alot within the swagger-ui to make various GET, POST, PUT actions. However, my preferred tool is POSTMAN. I needed a way to figure out how to get all the swagger-ui converted to POSTMAN. Upon searching, I came across this method mentioned here.

To fetch all the swagger JSON output, go to this URL on the VMware UAG Appliance.

https://uagnameorip:9443/rest/swagger.json

We have two options here. #Option1 – copy all the data from the webpage and paste it under Postman – Import – Paste Raw Text. You will have all the VMware UAG Access Gateway Rest API listed. #Option2 – Paste the above URL into Postman – Import – Import from link (This didn’t work for me maybe authentication was required)

Postman – Import

Please find attached the POSTMAN export for the VMware Unified Access Gateway Appliance 3.9.1. (Note I believe swagger-ui was availble post UAG 3.7 onwards).

Postman – API Calls UAG

I hope you will find this post useful to start using the Swagger-UI and Postman collections to begin working with UAG appliance. My request if you further create interesting scripts or perform cool activities with UAG appliance, I hope you can share it back with me?

Thanks,
Aresh Sarkari

Tags: , , ,

Report all VMware App Volumes Writable Volumes with Status Disabled and Orphaned

22 Apr

Often within the App Volumes Manager, there are Writable Volumes that will show up as Status “Orphaned” and essentially that can be caused by active directory user accounts that have been disabled in AD.

Writable Status = Orphaned

There is also a Status called “Disabled” and that can be caused when an App Volumes administrator decides to disable the Writable Volumes.

Writable Status = Disabled

Now if you have a enteprise environment with 1000’s of users, it’s hard to perform this activity from the UI. I have created a script that can report on the status of “Orphaned” and “Disabled” send you the output in *.csv report on a daily/weekly basis as per your needs.

####################################################################
# Get List of Writable Volumes from AppVolumes Manager for Status Disabled and Orphaned
# Author - Aresh Sarkari (@askaresh)
# Version - V2.0
####################################################################

# Run at the start of each script to import the credentials
$Credentials = IMPORT-CLIXML "C:\Scripts\Secure-Creds\SCred_avmgr.xml"
$RESTAPIUser = $Credentials.UserName
$RESTAPIPassword = $Credentials.GetNetworkCredential().Password


$body = @{
    username = “$RESTAPIUser"
    password = “$RESTAPIPassword”
}

Invoke-RestMethod -SessionVariable DaLogin -Method Post -Uri "https://avolmanager.askaresh.com/cv_api/sessions” -Body $body

$output = Invoke-RestMethod -WebSession $DaLogin -Method Get -Uri "https://avolmanager.askaresh.com/cv_api/writables" -ContentType "application/json"

$output.datastores.writable_volumes | Select-Object owner_name, owner_upn, title, status | Where-Object {[string]$_.status -match "Orphaned" -and $_.title -match "(disabled)"} | Export-Csv -NoTypeInformation -Append D:\Aresh\Orphaned.Disabled-Writables.$(Get-Date -Format "yyyyMMddHHmm").csv

#send an email (provided the smtp server is reachable from where ever you are running this script)
$emailfrom = 'writablevolumes@askaresh.com'
$emailto = 'email1@askaresh.com', 'email2@askaresh.com'
$emailsub = 'Wrtiable Volumes with status Orphaned and Disabled - Weekly'
$emailbody = 'Attached CSV File from App Volumes Manager. The attachment included the API response for all the Writable which are orphaned and Disabled in UI'
$emailattach = "D:\Aresh\Orphaned.Disabled-Writables.$(Get-Date -Format "yyyyMMddHHmm").csv"
$emailsmtp = 'smtp.askaresh.com'

Send-MailMessage -From $emailfrom -To $emailto -Subject $emailsub -Body $emailbody -Attachments $emailattach -Priority High -DeliveryNotificationOption OnFailure -SmtpServer $emailsmtp

GitHub – https://github.com/askaresh/scripts/blob/master/wrtiable-orph-disa

Depending upon the output, you can have your service desk get in touch with the Active Directory teams to get the affected end-users to be removed from the App volumes writable volumes entitled groups and then proceed towards clean up of their writable volumes if there is no legal hold requirements.

I hope you will find this script useful to get a report for all writable volumes with status Orphaned and Disabled. My request if you further enhance the script or make it more creative, I hope you can share it back with me?

Thanks,
Aresh Sarkari

Report all VMware App Volumes Writable Volumes with low disk space

20 Apr

We have provided end-users with 30 GB Writable Volumes, and within the App Volumes Manager console there is an ability in the UI to see the Writable Volumes disk free under the view called – “Usage View”

Writable Volumes - Usage View
Writable Volumes – Usage View

The biggest challenge is if you have 1000’s of users, it’s hard to perform this activity from the UI. I have created a script that can send you the output in *.csv report on a daily/weekly basis as per your needs.

####################################################################
# Get List of Wrtiable Volumes from AppVolumes Manager for free space less than 3 GB out of 30 GB
# Author - Aresh Sarkari (@askaresh)
# Version - V2.0
####################################################################


# Run at the start of each script to import the credentials
$Credentials = IMPORT-CLIXML "C:\Scripts\Secure-Creds\SCred_avmgr.xml"
$RESTAPIUser = $Credentials.UserName
$RESTAPIPassword = $Credentials.GetNetworkCredential().Password


$body = @{
    username = “$RESTAPIUser"
    password = “$RESTAPIPassword”
}

Invoke-RestMethod -SessionVariable DaLogin -Method Post -Uri "https://avolmanager.askaresh.com/cv_api/sessions” -Body $body

$output = Invoke-RestMethod -WebSession $DaLogin -Method Get -Uri "https://avolmanager.askaresh.com/cv_api/writables" -ContentType "application/json"

$output.datastores.writable_volumes | Select-Object owner_name, owner_upn,total_mb, free_mb, percent_available, status | Where-Object {$_.free_mb -lt 3072}  | Export-Csv -NoTypeInformation -Append D:\Aresh\Writableslt3gb.$(Get-Date -Format "yyyyMMddHHmm").csv

#send an email (provided the smtp server is reachable from where ever you are running this script)
$emailfrom = 'writablevolumes@askaresh.com'
$emailto = 'email1@askaresh.com', 'email2@askaresh.com' #Enter your SMTP Details
$emailsub = 'Wrtiable Volumes Size (free_mb) less than 3 GB out of 30 GB - 24 Hours'
$emailbody = 'Attached CSV File from App Volumes Manager. The attachment included the API response for all the Writable Volumes less than 3 GB of free space'
$emailattach = "D:\Aresh\Writableslt3gb.$(Get-Date -Format "yyyyMMddHHmm").csv"
$emailsmtp = 'smtp.askaresh.com'

Send-MailMessage -From $emailfrom -To $emailto -Subject $emailsub -Body $emailbody -Attachments $emailattach -Priority High -DeliveryNotificationOption OnFailure -SmtpServer $emailsmtp

GitHub – https://github.com/askaresh/scripts/blob/master/writablevolumesdiskusage

Depending upon the output, you can have your service desk get in touch with the affected end-users to clear-up disk space or provide options for further expansion.

I hope you will find this script useful to get a report for all writable volumes nearing their disk space usage. My request if you further enhance the script or make it more creative, I hope you can share it back with me?

Thanks,
Aresh Sarkari

Intermittent Clipboard issues on VMware Horizon virtual desktop

18 Apr

Recently, we had an issue within our environment where-in end-users complained of intermittently one-way clipboard not working(Virtual Desktop to Endpoint will fail). The tricky part here was it would happen intermittently to anyone without any set pattern.

Environment Details:
VMware Horizon 7.11
VMware App Volumes 2.18.1
VMware Dynamic Environment Manager 9.10
VMware Horizon Client 5.x

Process of elimination

  • We were not using the Horizon Blast GPO for setting the clipboard.
  • The clipboard was setup using DEM Horizon Smart Policies – Enabled Both Directions
  • Upgrade the Horizon Client to the latest version to remove any Client related issues
  • We already had the anti-virus process exclusion of VMwareViewClipboard.exe
  • We disabled the Writable Volumes, and the clipboard issue will never occur.

Resolution

The above test made it evident that something within the Writable Volumes was causing the intermittent clipboard issue. The next thing that came to mind is adding path/process exclusion within the snapvol.cfg. One may ask how did you determine that path, but recently we have had many application issues that needed exclusion to make them work.

What I didn’t know was which path or process, until the task manager showed a clipboard process for Horizon called – VMwareViewClipboard.exe and its Path – C:\Program Files\Common Files\VMware\Remote Experience\x64. I read many communities post having mentioned this process. However, I wasn’t sure if adding the entire path exclusion made sense as I could see many Horizon process *.exe and wasn’t sure what additional repercussions it can have. I went ahead, adding the below process exclusion.

exclude_process_name=VMwareViewClipboard.exe
Process exclusion in writable volumes snapvol.cfg

Post adding the exclusion, all the end-users with intermittent clipboard issues were always able to do two side clipboard. In this blog, I am not outlining the steps on how to add the snapvol.cfg exclusion as my ex-colleague Daniel Bakshi outlines on a VMware blog post on how to do it step by step.

Update 2nd May 2020
We had a VMware GSS support case open on the same issue, and they came back with a suggestion to exclude this registry path instead of the process exclusions. Note we been told there is no impact with process or registry, but its a good practice to do registry/path exclusions instead of the process. This registry/subkeys are responsible for the Clipboard – DEM Horizon Smart Policies.

exclude_registry=\REGISTRY\MACHINE\SOFTWARE\VMware, Inc.\VMware UEM
Process exclusion in writable volumes snapvol.cfg

I hope you will find this information useful if you encounter intermittent clipboard issues.

Thanks,
Aresh Sarkari

Tags: , , ,

Black Screen on mobile devices during logon – VMware Horizon and VMware Workspace One

17 Dec

We had a strange issue in which end-users reported a black screen when they clicked on their Desktop tile in Workspace One portal on their mobile devices on Android and iOS. The moment they clicked on the endpoint the black screen would go away and it would give the logon banner and normal Windows 10 logon.

Usual Suspects

Our investigation led to Windows Logon Banner applied via the group policy causing the black screen. We were soon able to rule out by disabling the logon banner and the black screen persisted.
The black screen only appear on mobile devices. The Desktop/Laptops you didnt observe the issue.

EUC Stack

VMware Horizon 7.6
VMware App Volumes 2.14.2
VMware Identity Manager 3.3
VMware User Environment Manager 9.4
Windows 10 1803

Resolution

We managed to open the VMware GSS case and a lot of troubleshooting was carried out from re-running the VMware OSOT tool and changing the Power Configuration policy.

The final configuration that resolved the black screen issue:

Open the master image and run PowerShell with administrative rights and execute the following commands:

powercfg -change -monitor-timeout-ac 0
powercfg -change -monitor-timeout-dc 0

(Note – Here 0 means Never)

ScreenSettings

Power and Screen Settings – Windows 10

Make sure you restart the master template post implementing the commands . Take a snapshot and perform “Push-Image” operation in Horizon Administror console.

I hope you will find this information useful if you encounter the Black Screen issue. A big thanks to Manivannan Arul my teammate for his continoursly effort while troubleshooting with GSS.

Thanks,
Aresh Sarkari

Continue reading →

Tags: ,

VMware EUC – Horizon, UAG, VIDM and AppVolumes – NSX Load Balancing – Health Check Monitors

2 Feb

There is no single place to find a consolidated list of Load balancer health check monitors (aka Service Monitors in NSX) for the VMware EUC products:

I have been using VMware NSX load balancer across the board. The below details will provide an overview of what to enter for the health monitors. Note – If you are using something more meaningful  for your environment leave feedback in the comments section. I will try to implement the same and update the blog later.

VMware Unified Access Gateway (UAG)

Create a new Service Monitor under NSX and call is UAG_https_monitor. Refer to the screenshot for more details.

UAG Service Monitor

Send String: GET /favicon.ico
Send String: GET /favicon.icoSend String: GET /favicon.ico
Send String: GET /favicon.ico
Send String: GET /favicon.ico

VMware Identity Manager (VIDM)

Create a new Service Monitor under NSX and call is VIDM_https_monitor. Refer to the screenshot for more details.

VIDM Service Monitor
Send String: GET /SAAS/auth/login
Response code: 200s

Horizon Connection Servers

Create a new Service Monitor under NSX and call is Horizon_https_monitor. Refer to the screenshot for more details.

image
Send String: GET /broker/xml/
Receive string: /styles/clientlaunch-default

VMware App Volumes

Create a new Service Monitor under NSX and call is AV_https_monitor. Refer to the screenshot for more details.

AV Service Monitor

I hope you will find these monitors useful in monitoring the VMware EUC products.

Thanks,
Aresh Sarkari


Bye Bye VMware and Hello Dell EMC here I come!

16 Aug

Dell EMC

On 9th Aug 2018, I took the most difficult decisions of my life to leave my country India and one of the best companies in information technology VMware after working for 4.5 years. Our search for a better standard of living in a developed country and secured future for my kids ended up in Australia. We did evaluate Canada & Germany as well, but looking at all the factors, Australia was the place which we locked down to migrate and settle.

Back in early June, I started looking for an internal transfer within VMware. However, after talking to multiple bosses and divisions, nothing was materializing for an internal move. I had to make a hard choice to leave VMware and move ahead with my decision of moving countries. I began my job hunt during my notice period within VMware and 15 days later I came across a wonderful opportunity in Dell EMC @ Sydney, Australia. I am pleased that I got an opportunity in the parent and consortium of companies

Although I am going to miss working with a lot of colleagues @ VMware. But I am looking forward to working with new colleagues, projects and challenges in Dell EMC. It would be fun learning new things and solving problems from the real world and get an outside perspective. Will keep in touch on Twitter and LinkedIn

Thanks,
Aresh Sarkari

Update VMware vSAN Storage Controller Firmware and Driver – In three easy steps

9 Jul

We were having frequent hardware issues on our Dell PowerEdge R630 hyper converged servers and the Dell Support recommended in upgrading the Avago (LSI) Dell PERC H730 Mini Controller Driver and Firmware to the latest version.

Avago (LSI) Dell PERC H730 Mini

Existing Version

Dell Recommended

Firmware

25.5.2.0001

25.5.4.0006

Driver

6.910.18.00

7.703.18.00-1OEM

Note: The procedure of upgrade is applicable and tested on vSphere 6.5 U1 or vSAN 6.6 environments

We were running the support Driver/Firmware as all vSAN Health checks were green. However, there was a latest build available which included few additional fixes. Let me show you how easy it was to update the Storage Controller Drivers and Firmware using the VMware Update Manager and vSAN Configuration Assistant Upgrade Tool.

The Sequence that we need to execute is as follows:

  • Upgrade the Driver – LSI_mr3 VIB driver using the VMware Update Manager
  • After doing the above step you will start seeing the Controller Firmware and Avago Management Tool listed in the vSAN Configuration Assistant Update Tool

Step 1 – Sync Cluster to Perform Online Test

Add the my.vmware.com account in the vCenter to enable checking/sync with the online engine

  • Enter the Username and Password
vSAN Build Recommendation Engine

Step 2 – Update Storage Controller Driver

The VMware Update Manager (VUM) will perform the Controller Driver Update in rolling reboot fashion one ESXi Host at a time

  • Select the Cluster and Choose the lsi_mr3: Avago Native MegaRAID SAS driver
Update Manager - LSI Controller Driver Package
  • Click on the Remediate Button
    • Select VSAN Cluster under Baseline Groups and the VIB Driver LSI under Baselines
    • Click Next
VUM Baselines Selection
  • Select all the Host in the Cluster (E.g. If you want to perform a quick test you can select one-host). In our case we selected all the 21 hosts
VUM - Select the Host

  • Select the Package
VUM - Select the Package

  • Click on Ignore warning
VUM - Ignore the warning

  • Select Do Not Change Power State and leave the timings to defaults
VUM - Power State

  • Select the three options as below
    • Disable DPM
    • Disable HA admission control
    • Migrate Powered-off VM
VUM - Cluster Remediation Options

  • Click Finish
    • Click on Pre-check Remediation and see the current configuration
    • The upgrade will start in a rolling reboot (1-by-1)
VUM - Configuration Preview

  • Make sure to verify the versions of the driver is showing updated/passed in the VSAN health tests

Step 3 – Storage Controller Firmware

Once the Driver is installed the Controller Management Tools and Firmware for Avago get listed together in the vSAN Configuration Assistant Update tool.

Both the components get installed onto the ESXi host together:

  • Click on Download and that will turn the status in Ready to install
  • Click on the Update All  and select the second option Rolling reboot (one host at a time)
  • Install the Firmware on all the ESXi Host within the cluster.
vSAN - Configuration Assistant Update Tool

  • Make sure to verify the versions of the firmware is showing updated/passed in the VSAN health tests

Check the Up-time of the ESXi host (Check every 2 hours and update the tracker)

This step will enable you to track the progress of the cluster as on how many host are done. In our scenario for the Driver (8 hr) and Controller (8 hr) combined together for a 21 host cluster it took close to 16 hours. Off course this number will vary depending upon the cluster usage.

  • Connect to the vCenter Linked mode

Connect-VIServer -Server rack-1-vc-5.domain.com -Protocol https -AllLinked

  • Check the Uptime of ESXi Host

Get-VMHost | Get-View | select Name, @{N="Uptime"; E={(Get-Date) - $_.Summary.Runtime.BootTime}}

I hope you will find these steps useful in upgrading the VSAN Controller/Driver firmware easily using the Update Manager + Configuration Assist Update Tool. Let me know, if you have additional questions in the comments section.

Thanks,
Aresh Sarkari

Poor man’s Samsung DEX HUB and VMware Horizon Advantage

22 Jun

I had been intrigued by the Samsung DeX mode post its launch but didn’t have the courage to buy the 125$ (Rs. 10,000/- INR) Samsung DeX Station. I was on a look-out for an alternate Hub which could do the DeX mode on my Galaxy S8+ for a lot less. After searching @ AliExpress I finally managed to find a hub called EASYA Thunderbolt 3 USB C to HDMI Adapter DeX Mode for Samsung Galaxy S8/S9 which had some good positive reviews and for 33$

The moment of truth was to plug-in the Galaxy S8+ and try the DeX mode. Attempt-1: Managed to plug the phone to the hub and HDMI monitor as the output. Next thing I noticed was the screen mirror got enabled and DeX Mode pop-up wouldn’t come or get detected.

Attempt-2: Additional to the above I plugged in the Power in the USB-C 3.1 PD Port and magically the pop-up appeared on the phone “Start DeX Mode”

If you don’t have the wireless mouse plugged in the entire Galaxy S8+ screen acts like a mouse trackpad which can come-in handy.

EASYA Thunderbolt 3 USB Type-C Hub To HDMI Adapter Dex Mode

Productivity with VMware Horizon:

The Horizon Client available on the Android Store has integration with DeX mode that enables you to use the Virtual Desktop in Full screen mode. I launch my Windows 10 Desktop and use it for an entire day. I was easily able to work on the following applications without any issues

  • Microsoft Outlook Client
  • Chrome and Firefox browser
  • Skype for Business (Audio/Chat Only) – Video was having issues
  • VMware Performance Tracker was showing the CPU and Network Bandwidth Usage graphs in real-time
  • There was no lag or any sign of slowness in any form
  • CPU Usage on the phone at an average of 4-6%

Known Observations:

  • The phone didn’t heat all day during its usage
  • The HUB was reasonably warm during the entire day usage
  • The limited DeX compatible Application works good in full-screen

More Picture on the Usage

DeX Mode and Horizon Client Launch
Horizon Client

DeX Mode and Full Screen – Windows 10 + Dell 24 inch Monitor
Full Screen - Windows 10 VDI

Hardware Setup – Logitech M140 Bluetooth Keyboard + Mouse
Hardware-Setup

More Documentation on Samsung DeX + VMware Horizon

Using Horizon Client with Samsung DeX
Enable the DeX Mode Auto Launch Feature

I hope you find this HUB review and DeX mode usage with Horizon useful and will be able to use it as a daily driver. Let me know if you would like to know more in the comments section

Thanks,
Aresh

← Older Entries