Script create read-only account for monitoring VMware Unified Access Gateway

23 Sep

We have been using VMware Unified Access Gateway (UAG) for quite a few years. To monitor the appliance using vROPS or other monitoring tools or API calls scripts you need a read-only monitoring account created in the console under “Account Settings”.

Account Settings - UAG
Read-only account for monitoring

In our deployment we have 14 UAG appliances (Internal/External) – Yes we tunnel internal connections too. Post the upgrade we had to re-create the read-only account for the API call monitoring on all 14 appliances. The following script I wrote to create the read-only account per UAG server. Just change the IP and point to another UAG to create accounts.

####################################################################
# Create ready-only account in the VMware Unified Access Gateway Appliance
# for monitoring purposes using vROPS or API etc.
# Author - Aresh Sarkari (@askaresh)
# Version - V5.0
####################################################################


# Ignore UAG cert errors (self signed or 

add-type @"
    using System.Net;
    using System.Security.Cryptography.X509Certificates;
    public class TrustAllCertsPolicy : ICertificatePolicy {
        public bool CheckValidationResult(
            ServicePoint srvPoint, X509Certificate certificate,
            WebRequest request, int certificateProblem) {
            return true;
        }
    }
"@
[System.Net.ServicePointManager]::CertificatePolicy = New-Object TrustAllCertsPolicy
[System.Net.ServicePointManager]::SecurityProtocol = [System.Net.SecurityProtocolType]'Ssl3,Tls,Tls11,Tls12'


##API Call to make the intial connection to the UAG Appliance##

$Uri = "https://10.0.0.1:9443/rest/v1/config/adminusers/logAdminUserAction/LOGIN"
$Username = "admin"
$Password = "adminpassword"

$Headers = @{ Authorization = "Basic {0}" -f [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(("{0}:{1}" -f $Username,$Password))) }

Invoke-RestMethod -SessionVariable DaLogin -Uri $Uri -Headers $Headers


###API Call to create the user account with read-only access under VMware Unified Access Gateway##

$body = @{
  name = "UAG_vRops"
  password= "typeyourpassword"
  enabled=$true
  roles = @("ROLE_MONITORING")
  noOfDaysRemainingForPwdExpiry=0
} | ConvertTo-Json

$output = Invoke-RestMethod -WebSession $DaLogin -Method Put -Uri "https://10.0.0.1:9443/rest/v1/config/adminusers" -Body $body -ContentType "application/json"

Write-Output $output

GitHub https://github.com/askaresh/scripts/blob/master/uagreadonlyacct

I hope you will find this script useful to create the UAG read only accounts and would not have to create them manually on multiple appliances. My request if you further enhance the script or make it more creative, I hope you can share it back with me?

Thanks,
Aresh Sarkari

Unable to uninstall/upgrade VMware Horizon Client within the VMware App Volumes AppStack

22 Jul

We had a very long ongoing issue wherein we couldn’t uninstall or upgrade the VMware Horizon Client within the AppStack. We had successfully installed the Horizon Client within the AppStack. However, when it was time to perform an upgrade or uninstall to the latest version, it would fail during a reboot with the following error.

Unknown HardError

We initially saw the issue on App Volumes 2.14. While we were troubleshooting for an extended period, we upgrade to App Volumes 2.18.1, and both the versions exhibited the same failure during uninstall or upgrade.

Process to reproduce the error

  • Upgrade horizon client –> reboot –> hard error
  • Uninstall horizon client –> reboot –>hard error
  • Uninstall horizon client –> install horizon client –> reboot –> hard error
  • Upgrade horizon client –> complete provisioning without reboot –> completes successful –> during next update of AppStack it crashes with Hard error
  • Uninstall horizon client –> complete provisioning without reboot –> completes successful –> during next update of AppStack it crashes with Hard error

Environment Details

VMware Horizon 7.11
VMware App Volumes 2.18.1
VMware Dynamic Environment Manager 9.10
VMware Horizon Client 5.x

Process of elimination

  • Upgrade the Horizon Client to the various 5.x version to remove any version specific Client related issues
  • We didn’t have Antivirus running on the AppStack capturing template
  • We could build the AppStack from scratch with the newer version of Horizon Client but only upgrade/uninstall would fail
  • We were honestly running out of troubleshooting ideas

Resolution

After trying out all the usual steps and avoid re-creating AppStack every single time during life cycle management, we managed to open a VMware GSS case handled by Karan Ahuja(Very helpful support engineer), which ended been worked by the engineering team(Art Rothstein – Champ in AV Eng Team). Note quite alot of logs and Procmon were exchanged from the problematic application capturing VM template. Finally, the fix was determined as a AppStack snapvol.cfg exclusion. After putting this exclusion into the AppStack – App capturing VM during provisioning we could upgrade or uninstall Horizon Client.

exclude_registry=\REGISTRY\MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\ProfileService
Path exclusion in AppStack snapvol.cfg

Disclaimer – Due to the nature of the issue and time taken to resolve it we decided to move the Horizon Client from AppStack into the base image. However, the fix is validated, and 100% working post the exclusion.

I hope you will find this information useful if you encounter the issue. A big thanks to Manivannan Arul my teammate for his continuous effort while troubleshooting with GSS over a period of 4+ months.

Thanks,
Aresh Sarkari

Black screen when re-connect to VMware Horizon virtual desktop

27 May

We had an issue after we upgraded our EUC Stack, especially VMware App Volumes 2.14 to 2.18.1. Quite a few end-users started reporting black screen when they were trying to re-connect to their desktops post the original session launch. This would mean re-connect post breaks, endpoint screen locks, next working day re-connections, etc.

EUC Environment Details:
VMware Horizon 7.11
VMware App Volumes 2.18.1
VMware Dynamic Environment Manager 9.10
VMware Horizon Client 5.x
VMware Workspace One 3.3

Process of elimination

  • If we re-created the writable volumes of the problematic end-users the black screen issue would go away. This provided us with a clue that the problem lied with VMware App Volumes – Writable Volumes
  • No errors/failures observed within the VMware DEM/Horizon logs
  • Upgrade the Horizon Client to the latest 5.x version to remove any Client related issues
  • We already had the necessary anti-virus exclusion based on VMware Antivirus Considerations in a VMware Horizon 7 Environment

Resolution
After trying out all the usual steps and avoid re-creating writable volumes for problematic end-users, we managed to open a VMware GSS case handled by Karan Ahuja(Very helpful support engineer), which ended been worked by the engineering team(Art Rothstein – Champ in AV Eng Team). Note quite alot of logs, memory dumps, and Procmon were exchanged from the problematic VM using various remote gathering techniques. Finally, the fix was determined as a writable volume snapvol.cfg exclusion. (In our case, the problem is caused by smss.exe using a copy of winlogon.exe that is on the writable volume). After putting this exclusion into all problematic end-users, they stopped seeing Black screen issues upon re-connect.

exclude_path=%SystemRoot%\System32\winlogon.exe
Path exclusion in writable volumes snapvol.cfg

In this blog, I am not outlining the steps on how to add the snapvol.cfg exclusion as my ex-colleague Daniel Bakshi outlines on a VMware blog post on how to do it step by step. I hope you will find this information useful if you encounter intermittent black screen issues.

Thanks,
Aresh Sarkari

Swagger-UI and Postman Collection for VMware Unified Access Gateway

6 May

I aimed to perform a particular VMware Unified Access Gateway (UAG) tasks programatically. After some guidance from Mark Benson he introduced me to the Swagger-UI that is available within the product.

To access the Swagger-UI on UAG open the following URL within the browser and enter your username and password.

https://uagnameorip:9443/swagger-ui/index.html
Swagger-UI – UAG API Calls

One can do alot within the swagger-ui to make various GET, POST, PUT actions. However, my preferred tool is POSTMAN. I needed a way to figure out how to get all the swagger-ui converted to POSTMAN. Upon searching, I came across this method mentioned here.

To fetch all the swagger JSON output, go to this URL on the VMware UAG Appliance.

https://uagnameorip:9443/rest/swagger.json

We have two options here. #Option1 – copy all the data from the webpage and paste it under Postman – Import – Paste Raw Text. You will have all the VMware UAG Access Gateway Rest API listed. #Option2 – Paste the above URL into Postman – Import – Import from link (This didn’t work for me maybe authentication was required)

Postman – Import

Please find attached the POSTMAN export for the VMware Unified Access Gateway Appliance 3.9.1. (Note I believe swagger-ui was availble post UAG 3.7 onwards).

Postman – API Calls UAG

I hope you will find this post useful to start using the Swagger-UI and Postman collections to begin working with UAG appliance. My request if you further create interesting scripts or perform cool activities with UAG appliance, I hope you can share it back with me?

Thanks,
Aresh Sarkari

Report all VMware App Volumes Writable Volumes with Status Disabled and Orphaned

22 Apr

Often within the App Volumes Manager, there are Writable Volumes that will show up as Status “Orphaned” and essentially that can be caused by active directory user accounts that have been disabled in AD.

Writable Status = Orphaned

There is also a Status called “Disabled” and that can be caused when an App Volumes administrator decides to disable the Writable Volumes.

Writable Status = Disabled

Now if you have a enteprise environment with 1000’s of users, it’s hard to perform this activity from the UI. I have created a script that can report on the status of “Orphaned” and “Disabled” send you the output in *.csv report on a daily/weekly basis as per your needs.

####################################################################
# Get List of Writable Volumes from AppVolumes Manager for Status Disabled and Orphaned
# Author - Aresh Sarkari (@askaresh)
# Version - V2.0
####################################################################

# Run at the start of each script to import the credentials
$Credentials = IMPORT-CLIXML "C:\Scripts\Secure-Creds\SCred_avmgr.xml"
$RESTAPIUser = $Credentials.UserName
$RESTAPIPassword = $Credentials.GetNetworkCredential().Password


$body = @{
    username = “$RESTAPIUser"
    password = “$RESTAPIPassword”
}

Invoke-RestMethod -SessionVariable DaLogin -Method Post -Uri "https://avolmanager.askaresh.com/cv_api/sessions” -Body $body

$output = Invoke-RestMethod -WebSession $DaLogin -Method Get -Uri "https://avolmanager.askaresh.com/cv_api/writables" -ContentType "application/json"

$output.datastores.writable_volumes | Select-Object owner_name, owner_upn, title, status | Where-Object {[string]$_.status -match "Orphaned" -and $_.title -match "(disabled)"} | Export-Csv -NoTypeInformation -Append D:\Aresh\Orphaned.Disabled-Writables.$(Get-Date -Format "yyyyMMddHHmm").csv

#send an email (provided the smtp server is reachable from where ever you are running this script)
$emailfrom = 'writablevolumes@askaresh.com'
$emailto = 'email1@askaresh.com', 'email2@askaresh.com'
$emailsub = 'Wrtiable Volumes with status Orphaned and Disabled - Weekly'
$emailbody = 'Attached CSV File from App Volumes Manager. The attachment included the API response for all the Writable which are orphaned and Disabled in UI'
$emailattach = "D:\Aresh\Orphaned.Disabled-Writables.$(Get-Date -Format "yyyyMMddHHmm").csv"
$emailsmtp = 'smtp.askaresh.com'

Send-MailMessage -From $emailfrom -To $emailto -Subject $emailsub -Body $emailbody -Attachments $emailattach -Priority High -DeliveryNotificationOption OnFailure -SmtpServer $emailsmtp

GitHub – https://github.com/askaresh/scripts/blob/master/wrtiable-orph-disa

Depending upon the output, you can have your service desk get in touch with the Active Directory teams to get the affected end-users to be removed from the App volumes writable volumes entitled groups and then proceed towards clean up of their writable volumes if there is no legal hold requirements.

I hope you will find this script useful to get a report for all writable volumes with status Orphaned and Disabled. My request if you further enhance the script or make it more creative, I hope you can share it back with me?

Thanks,
Aresh Sarkari

Report all VMware App Volumes Writable Volumes with low disk space

20 Apr

We have provided end-users with 30 GB Writable Volumes, and within the App Volumes Manager console there is an ability in the UI to see the Writable Volumes disk free under the view called – “Usage View”

Writable Volumes - Usage View
Writable Volumes – Usage View

The biggest challenge is if you have 1000’s of users, it’s hard to perform this activity from the UI. I have created a script that can send you the output in *.csv report on a daily/weekly basis as per your needs.

####################################################################
# Get List of Wrtiable Volumes from AppVolumes Manager for free space less than 3 GB out of 30 GB
# Author - Aresh Sarkari (@askaresh)
# Version - V2.0
####################################################################


# Run at the start of each script to import the credentials
$Credentials = IMPORT-CLIXML "C:\Scripts\Secure-Creds\SCred_avmgr.xml"
$RESTAPIUser = $Credentials.UserName
$RESTAPIPassword = $Credentials.GetNetworkCredential().Password


$body = @{
    username = “$RESTAPIUser"
    password = “$RESTAPIPassword”
}

Invoke-RestMethod -SessionVariable DaLogin -Method Post -Uri "https://avolmanager.askaresh.com/cv_api/sessions” -Body $body

$output = Invoke-RestMethod -WebSession $DaLogin -Method Get -Uri "https://avolmanager.askaresh.com/cv_api/writables" -ContentType "application/json"

$output.datastores.writable_volumes | Select-Object owner_name, owner_upn,total_mb, free_mb, percent_available, status | Where-Object {$_.free_mb -lt 3072}  | Export-Csv -NoTypeInformation -Append D:\Aresh\Writableslt3gb.$(Get-Date -Format "yyyyMMddHHmm").csv

#send an email (provided the smtp server is reachable from where ever you are running this script)
$emailfrom = 'writablevolumes@askaresh.com'
$emailto = 'email1@askaresh.com', 'email2@askaresh.com' #Enter your SMTP Details
$emailsub = 'Wrtiable Volumes Size (free_mb) less than 3 GB out of 30 GB - 24 Hours'
$emailbody = 'Attached CSV File from App Volumes Manager. The attachment included the API response for all the Writable Volumes less than 3 GB of free space'
$emailattach = "D:\Aresh\Writableslt3gb.$(Get-Date -Format "yyyyMMddHHmm").csv"
$emailsmtp = 'smtp.askaresh.com'

Send-MailMessage -From $emailfrom -To $emailto -Subject $emailsub -Body $emailbody -Attachments $emailattach -Priority High -DeliveryNotificationOption OnFailure -SmtpServer $emailsmtp

GitHub https://github.com/askaresh/scripts/blob/master/writablevolumesdiskusage

Depending upon the output, you can have your service desk get in touch with the affected end-users to clear-up disk space or provide options for further expansion.

I hope you will find this script useful to get a report for all writable volumes nearing their disk space usage. My request if you further enhance the script or make it more creative, I hope you can share it back with me?

Thanks,
Aresh Sarkari

Intermittent Clipboard issues on VMware Horizon virtual desktop

18 Apr

Recently, we had an issue within our environment where-in end-users complained of intermittently one-way clipboard not working(Virtual Desktop to Endpoint will fail). The tricky part here was it would happen intermittently to anyone without any set pattern.

Environment Details:
VMware Horizon 7.11
VMware App Volumes 2.18.1
VMware Dynamic Environment Manager 9.10
VMware Horizon Client 5.x

Process of elimination

  • We were not using the Horizon Blast GPO for setting the clipboard.
  • The clipboard was setup using DEM Horizon Smart Policies – Enabled Both Directions
  • Upgrade the Horizon Client to the latest version to remove any Client related issues
  • We already had the anti-virus process exclusion of VMwareViewClipboard.exe
  • We disabled the Writable Volumes, and the clipboard issue will never occur.

Resolution

The above test made it evident that something within the Writable Volumes was causing the intermittent clipboard issue. The next thing that came to mind is adding path/process exclusion within the snapvol.cfg. One may ask how did you determine that path, but recently we have had many application issues that needed exclusion to make them work.

What I didn’t know was which path or process, until the task manager showed a clipboard process for Horizon called – VMwareViewClipboard.exe and its Path – C:\Program Files\Common Files\VMware\Remote Experience\x64. I read many communities post having mentioned this process. However, I wasn’t sure if adding the entire path exclusion made sense as I could see many Horizon process *.exe and wasn’t sure what additional repercussions it can have. I went ahead, adding the below process exclusion.

exclude_process_name=VMwareViewClipboard.exe
Process exclusion in writable volumes snapvol.cfg

Post adding the exclusion, all the end-users with intermittent clipboard issues were always able to do two side clipboard. In this blog, I am not outlining the steps on how to add the snapvol.cfg exclusion as my ex-colleague Daniel Bakshi outlines on a VMware blog post on how to do it step by step.

Update 2nd May 2020
We had a VMware GSS support case open on the same issue, and they came back with a suggestion to exclude this registry path instead of the process exclusions. Note we been told there is no impact with process or registry, but its a good practice to do registry/path exclusions instead of the process. This registry/subkeys are responsible for the Clipboard – DEM Horizon Smart Policies.

exclude_registry=\REGISTRY\MACHINE\SOFTWARE\VMware, Inc.\VMware UEM
Process exclusion in writable volumes snapvol.cfg

I hope you will find this information useful if you encounter intermittent clipboard issues.

Thanks,
Aresh Sarkari

Black Screen on mobile devices during logon – VMware Horizon and VMware Workspace One

17 Dec

We had a strange issue in which end-users reported a black screen when they clicked on their Desktop tile in Workspace One portal on their mobile devices on Android and iOS. The moment they clicked on the endpoint the black screen would go away and it would give the logon banner and normal Windows 10 logon.

Usual Suspects

Our investigation led to Windows Logon Banner applied via the group policy causing the black screen. We were soon able to rule out by disabling the logon banner and the black screen persisted.
The black screen only appear on mobile devices. The Desktop/Laptops you didnt observe the issue.

EUC Stack

VMware Horizon 7.6
VMware App Volumes 2.14.2
VMware Identity Manager 3.3
VMware User Environment Manager 9.4
Windows 10 1803

Resolution

We managed to open the VMware GSS case and a lot of troubleshooting was carried out from re-running the VMware OSOT tool and changing the Power Configuration policy.

The final configuration that resolved the black screen issue:

Open the master image and run PowerShell with administrative rights and execute the following commands:

powercfg -change -monitor-timeout-ac 0
powercfg -change -monitor-timeout-dc 0

(Note – Here 0 means Never)

ScreenSettings

Power and Screen Settings – Windows 10

Make sure you restart the master template post implementing the commands . Take a snapshot and perform “Push-Image” operation in Horizon Administror console.

I hope you will find this information useful if you encounter the Black Screen issue. A big thanks to Manivannan Arul my teammate for his continoursly effort while troubleshooting with GSS.

Thanks,
Aresh Sarkari

Continue reading

VMware EUC – Horizon, UAG, VIDM and AppVolumes – NSX Load Balancing – Health Check Monitors

2 Feb

There is no single place to find a consolidated list of Load balancer health check monitors (aka Service Monitors in NSX) for the VMware EUC products:

I have been using VMware NSX load balancer across the board. The below details will provide an overview of what to enter for the health monitors. Note – If you are using something more meaningful  for your environment leave feedback in the comments section. I will try to implement the same and update the blog later.

VMware Unified Access Gateway (UAG)

Create a new Service Monitor under NSX and call is UAG_https_monitor. Refer to the screenshot for more details.

UAG Service Monitor

Send String: GET /favicon.ico
Send String: GET /favicon.icoSend String: GET /favicon.ico
Send String: GET /favicon.ico
Send String: GET /favicon.ico

VMware Identity Manager (VIDM)

Create a new Service Monitor under NSX and call is VIDM_https_monitor. Refer to the screenshot for more details.

VIDM Service Monitor
Send String: GET /SAAS/auth/login
Response code: 200s

Horizon Connection Servers

Create a new Service Monitor under NSX and call is Horizon_https_monitor. Refer to the screenshot for more details.

image
Send String: GET /broker/xml/
Receive string: /styles/clientlaunch-default

VMware App Volumes

Create a new Service Monitor under NSX and call is AV_https_monitor. Refer to the screenshot for more details.

AV Service Monitor

I hope you will find these monitors useful in monitoring the VMware EUC products.

Thanks,
Aresh Sarkari


Bye Bye VMware and Hello Dell EMC here I come!

16 Aug

Dell EMC

On 9th Aug 2018, I took the most difficult decisions of my life to leave my country India and one of the best companies in information technology VMware after working for 4.5 years. Our search for a better standard of living in a developed country and secured future for my kids ended up in Australia. We did evaluate Canada & Germany as well, but looking at all the factors, Australia was the place which we locked down to migrate and settle.

Back in early June, I started looking for an internal transfer within VMware. However, after talking to multiple bosses and divisions, nothing was materializing for an internal move. I had to make a hard choice to leave VMware and move ahead with my decision of moving countries. I began my job hunt during my notice period within VMware and 15 days later I came across a wonderful opportunity in Dell EMC @ Sydney, Australia. I am pleased that I got an opportunity in the parent and consortium of companies

Although I am going to miss working with a lot of colleagues @ VMware. But I am looking forward to working with new colleagues, projects and challenges in Dell EMC. It would be fun learning new things and solving problems from the real world and get an outside perspective. Will keep in touch on Twitter and LinkedIn

Thanks,
Aresh Sarkari