Upgrade from VMware Horizon 7.13.1 to 8.6 (2206) fails with VMware Horizon View Blast Secure Gateway (VMBlastSG) could not be installed

22 Dec

Many customers are already in the process of upgrading from VMware Horizon 7.x to 8.x or will soon upgrade as the End Of Life dates are upcoming in April 2023. I want to share a rare experience wherein the Horizon upgrade from 7.13.1 to 8.6 version failed. In the rare occasion where in upgrade fails in the below mentioned manner, the workaround steps will come in handy.

We have only received the workaround from VMware support, and I intend to update the post once I get a complete RCA. At least the workaround can help someone not have to revert the entire environment instead, follow the workaround and avoid a lot of rework.

Issue

During the upgrade of the first connection server in the POD1, we encountered the following error five mins into the upgrade. Note before starting the upgrade, the entire health dashboard for the POD was green and included backup and snapshots.

Environment Overview

Let take a look at the environment details to provide an high-level overview:
Active Site (POD1)

  • 5 VMware Horizon Connection Servers 7.13.1
  • SQL Database on Microsoft SQL 2016 Always-on – EventsDB
  • The 5 Brokers are behind an NSX Load balancer

Active Site (POD2)

  • 5 VMware Horizon Connection Servers 7.13.1
  • SQL Database on Microsoft SQL 2016 Always-on – EventsDB
  • The 5 Brokers are behind an NSX Load balancer

Observations

The logs from the installer had the following message it’s complaining about insufficient privileges, which the installer is run with admin privileges already.

MSI (s) (A0:E0) [06:12:07:977]: Executing op: ActionStart(Name=InstallServices,Description=Installing new services,Template=Service: [2])
Action 6:12:07: InstallServices. Installing new services
MSI (s) (A0:E0) [06:12:07:977]: Executing op: ProgressTotal(Total=9,Type=1,ByteEquivalent=1300000)
MSI (s) (A0:E0) [06:12:07:977]: Executing op: ServiceInstall(Name=PCOIPSG,DisplayName=VMware Horizon View PCoIP Secure Gateway,ImagePath="C:\Program Files\VMware\VMware View\Server\bin\SecurityGateway.exe",ServiceType=16,StartType=3,ErrorControl=1,,Dependencies=[~],,,Password=**********,Description=Provides VMware Horizon View PCoIP gateway services.,,)
InstallServices: Service:
MSI (s) (A0:E0) [06:12:08:228]: Executing op: ServiceInstall(Name=VMBlastSG,DisplayName=VMware Horizon View Blast Secure Gateway,ImagePath="C:\Program Files\VMware\VMware View\Server\appblastgateway\nssm.exe",ServiceType=16,StartType=3,ErrorControl=0,,Dependencies=[~],,,Password=**********,Description=Provides VMware Horizon View Blast gateway services.,,)
InstallServices: Service:
Info 1923.Service VMware Horizon View Blast Secure Gateway (VMBlastSG) could not be installed. Verify that you have sufficient privileges to install system services.
Action ended 6:12:08: InstallFinalize. Return value 3.

Also noticed during the upgrade, it should uninstall the services from version 7.13.1, and new services would be created for version 8.6. In this case, they were listed as disabled.

Workaround

Later during the ongoing RCA investigation, a workaround provided was working well. The reason for socializing the workaround is that we spent a tremendous amount of time and effort in a revert operation, which was very time consuming and cumbersome. Only if we had known about this workaround during the failed upgrade would we have saved tremendous effort.

#ProTip – If you have two POD, make sure before you start the upgrade. Take a snapshot of all the PODS together at the same time. This could help you in scenarios where the POD1 upgrade fails, and you can revert the entire environment (POD1 & POD2) from snapshots.

Prerequisite / Rollback Plan 

  1. Power off all the Connection Servers part of Cloud POD Federation 
  2. Take a powered-off snapshot
  3. In case of an incident during the change activity which requires recovering Horizon Environment, Snapshots will be used as a fallback plan as the last option if break/fix, troubleshooting steps performed is not resolving the issue. 
  4. In troubleshooting scenarios of a failed upgrade 

Workaround Steps 

  1. The following steps need to be performed on each Connection Server at a time 
  2. Uninstall HTML and Horizon Connection Server 7.13.1 component keeping ADLDS and ADLDSG Instances intact. Check services.msc, and Horizon Services will appear in the Disabled state 
  3. Perform a reboot of the Connection Server, and Horizon Services will be cleared from services.msc
  4. Install Horizon 8.6 as Standard, which will pick the residing ADLDS and ADLDSG instance
  5. After successful install, it can be verified using ldp utility as instructed in KB https://kb.vmware.com/s/article/2064157 and look for the fields whenChanged , and whenCreated . This step can be performed prior upgrade for comparing the state of ADLDS

The VMware GSS case handled by Jezill Asharaf (A very helpful support engineer) and a few of the backend engineering team has been instrumental. I hope you will find this information useful if you encounter the issue and it should help you save time. If you manage to tweak or improvise further on this solution, please don’t forget to keep me posted.

Thanks,
Aresh Sarkari

Leave a Reply

%d bloggers like this: