Archive | Windows RSS feed for this section

VMware Horizon Events Database – Annual Clean-up (purge old data)

14 May

VMware Horizon doesn’t restrict the growth of the historical tables in the Horizon Events database. VMware has a detailed knowledge base article with describes in details Purging old data from the View Events Database (2150309). However, there is a catch if you are trying to delete many records at one time, you will get transaction log full error. The below procedure will help you overcome the challenge. In our scenario, we purge the records once every year.

use HZNLOG
select count(*) from [dbo].[POD1_event_data_historical] where EventID in (select EventID from [dbo].[POD1_event_historical] where Time < '2021-01-31 00:00:00.000')
select count(*) from [dbo].[POD1_event_historical] where Time < '2021-01-31 00:00:00.000'

In the above example HZNLOG is the name of the database. POD1 is the prefix of the Horizon Events Database (Check in Horizon Admin console) and 2021-01-31 is the YYYY-MM-DD format (Show me all records before 31st Jan 2021)

No. of older records in Events DB

If we used the delete tables mentioned within the knowledge base article, we get the following error “The transaction log for database ‘HZNLOG’ is full due to ‘LOG_BACKUP”. Of course, the number of records in our case we are trying to delete is relatively high(Millions).

Error during deletion “Log is full”

You can shorten the above query for approx. 30 or 15 days, but still in our scenario, one would have to run the delete query more than 15 times to perform the annual clean-up. After searching around, I came across a blog post – Deleting millions of records from a table without blowing the transaction log (A big thank you Merill for sharing his knowledge) I tweaked it for my usecase of Horizon Events DB clean-up and, in a single query within 20 mins I could perform a yearly clean-up without any fuss of transaction log getting full. Essentially this performs the clean-up in a batch size of 10,000 row counts.

DECLARE @continue INT
DECLARE @rowcount INT
 
SET @continue = 1
WHILE @continue = 1
BEGIN
    PRINT GETDATE()
    SET ROWCOUNT 10000
    BEGIN TRANSACTION
	delete from [dbo].[POD1_event_data_historical] where EventID in (select EventID from [dbo].[POD1_event_historical] where Time < '2021-01-31 00:00:00.000')
	delete from [dbo].[POD1_event_historical] where Time < '2021-01-31 00:00:00.000'
    SET @rowcount = @@rowcount 
    COMMIT
    PRINT GETDATE()
    IF @rowcount = 0
    BEGIN
        SET @continue = 0
    END
END

The ouput will look something like below:

Enteire deletion in batches of 10K rows

After running the above deletion query, now re-run the select query to see if records exist before 31st Jan 2021, and now we have 0 records.

Zero records found

I hope you will find this SQL query helpful to perform Horizon Events Database clean-up in a jiffy. My request if you further enhance the query or make it more creative, I hope you can share it back with me?

Thanks,
Aresh Sarkari

VMware App Volumes – Volumes were not mounted due to an issue with your Writable Volume

18 Mar

Random floating desktop pools within our environment would exhibit issues where in the end-user would login to their desktop and they will be presented with a black screen with the message – Volumes were not mounted due to an issue with your Writable Volume. Please try logging in again, or contact your administrator.

Error

When this issue would surface, neither the AppStacks nor Writable Volumes would mount to the end-user desktop and if the end-user clicked on OK the session would log-off.

Environment Details

VMware Horizon 7.11
VMware App Volumes 2.18.5
VMware Dynamic Environment Manager 9.10
Windows 10 1909 Enterprise

Process of elimination

  • The App Volumes (AV) agent is able to communicate to the AV Manager on port 443 without any issues.
  • There were no SSL errors or load balancing issues communicating with the Agent/Manager.
  • We thought a particular Writable Volumes (WV) would be causing the issue. Deleted and re-created the WV still the issue would persist.
  • The issue would happen randomly for few users again and again.

Resolution

My team managed to open a VMware GSS case handled by Sanjay SP (A very helpful support engineer), he mentioned there were quite a few cases opened on a similar pattern. Following were the assessments from our logs:

  • During the first startup of Instant Clones, App Volumes Agent queries below registry key to know the customization status and updates manager with the same
    • [HKEY_LOCAL_MACHINE\SOFTWARE\VMware, Inc.\ViewComposer\ga\AgentIntegration]
      • “CustomizationState”=dword:1
  • It has a timeout of 300 seconds, and if this task times out AppVolumes manager will fail to create a unique identity for the VM in its database
  • In the App Volumes Agent logs, we see the respective timeout
    • [2021-03-09 07:11:34.009 UTC] [svservice:P1564:T1976] HandleNGVC: Waiting for NGVC to complete (count 299)
    • [2021-03-09 07:11:34.009 UTC] [svservice:P1564:T1976] Timed out waiting on NGVC after 300 seconds, disabling
  • The customization itself is working fine and we do see the registry entries getting updated with appropriate values. However, its not completed within 300 seconds. 

Fix

  • The delay in cloneprep customization was not found with IPv6 disabled on the primary nic adapter. The recommendation was to disable IPv6 since we don’t use it within the NIC adapter properties.
Disable IPv6 in the network adapters

I hope you will find this information useful if you encounter the issue. If you manage to tweak or improvise further on this solution, please don’t forget to keep me posted.

Thanks,
Aresh Sarkari

Internet Explorer crashing on Windows Server 2016 – Remote Desktop Session Host

18 Feb

We encountered a strange issue on the Windows Server 2016 Remote Desktop Session Host (RDSH) used for VMware Horizon Application Publishing. The Internet Explorer would launch and get into “Not Responding” state, and eventually, the process would close out without any errors.

IE Opening and Crashing

Process of elimination

  • We thought either Windows cumulative updates introduced the issue as it was working fine earlier.
  • There were no errors in the Windows Event Viewer (Application, System or Internet Explorer)
  • We used the Deployment Image Servicing and Management (DISM) command line tool to disable/enabled Internet Explorer without any luck.
    • dism /online /Disable-Feature /FeatureName:Internet-Explorer-Optional-amd64
    • dism /online /Enable-Feature /FeatureName:Internet-Explorer-Optional-amd64
  • Procmon is showing IE tries to launch the process multiple times, but the sub-process keep failing, and IE finally gives up at the end
IE Process launching multiple times
  • We were running out of troubleshooting ideas

Resolution

My team ended up opening a Microsoft Support case, and they could see that “Name Not Found for the ieproxy.dll” which is due to ieproxy.dll registration issues. Support confirmed they had seen similar instances in the past.

Please open command prompt with Admin rights and re-register the dll from System32 and Syswow64 folders.

%SystemRoot%\System32\regsvr32 ieproxy.dll

%SystemRoot%\Syswow64\regsvr32 ieproxy.dll

 I hope you will find this information useful if you encounter the issue. If you manage to tweak or improvise further on this solution, please don’t forget to keep me posted.

Thanks,
Aresh Sarkari

Horizon VDI – Calculator – Photos – Edge Not launching for end-users – Windows 10

8 Feb

In Windows 10 1909 VMware OST optimized image the end-users report they cannot open the following three built-in UWP windows application.

  • Microsoft Calculator
  • Microsoft Photos
  • Microsoft Edge browser

When the end-users try to open any of the three applications, nothing would happen – No error messages or pop-ups. The application doesn’t launch.

Environment Details

VMware Horizon 7.11
VMware App Volumes 2.18.5
VMware Dynamic Environment Manager 9.10

Process of elimination

  • The AppX package for (Calc, Photos and Edge) did exist in the base operating system
  • We can launch all the three applications within the optimized golden image template.
  • We were running the VMWare OSOT tool with the default VMware Windows 10 template. No additional customization or options selected.
  • One thing was evident the base template was working fine. The suspicion was around AppStack – App Volumes (We disabled the AppStacks/Writable Delivery – Same issue observed) or Dynamic Environment Manager causing the application from launching
  • We were running out of troubleshooting ideas

Resolution

Upon searching, I came across this community page – https://communities.vmware.com/t5/Horizon-Desktops-and-Apps/Windows-10-UWP-Applications-and-Taskbar/m-p/523086 and it outlined a solution of re-registering the UWP AppX package for the built-in application. We tried the fix in the DEV environment and it worked. Further it was replicated to the production setup.

Step 1: A Powershell script to register the AppX packages

Get-AppxPackage -allusers *windowscalculator* | Foreach {Add-AppxPackage -DisableDevelopmentMode -Register “$($_.InstallLocation)AppXManifest.xml”}
Get-AppxPackage -allusers *windows.photos* | Foreach {Add-AppxPackage -DisableDevelopmentMode -Register “$($_.InstallLocation)AppXManifest.xml”}
Get-AppXPackage -AllUsers *edge* | Foreach {Add-AppxPackage -DisableDevelopmentMode -Register "$($_.InstallLocation)AppXManifest.xml"}

Step 2 : Create a Dynamic Environment Manager – Logon Tasks

We selected to put the Powershell script within the UEM Share as the end-users have the read- access.

DEM - Logon Task
DEM-LogonTasks

 I hope you will find this information useful if you encounter the issue. If you manage to tweak or improvise further on this solution, please don’t forget to keep me posted.

Thanks,
Aresh Sarkari

Black screen when re-connect to VMware Horizon virtual desktop

27 May

We had an issue after we upgraded our EUC Stack, especially VMware App Volumes 2.14 to 2.18.1. Quite a few end-users started reporting black screen when they were trying to re-connect to their desktops post the original session launch. This would mean re-connect post breaks, endpoint screen locks, next working day re-connections, etc.

EUC Environment Details:
VMware Horizon 7.11
VMware App Volumes 2.18.1
VMware Dynamic Environment Manager 9.10
VMware Horizon Client 5.x
VMware Workspace One 3.3

Process of elimination

  • If we re-created the writable volumes of the problematic end-users the black screen issue would go away. This provided us with a clue that the problem lied with VMware App Volumes – Writable Volumes
  • No errors/failures observed within the VMware DEM/Horizon logs
  • Upgrade the Horizon Client to the latest 5.x version to remove any Client related issues
  • We already had the necessary anti-virus exclusion based on VMware Antivirus Considerations in a VMware Horizon 7 Environment

Resolution
After trying out all the usual steps and avoid re-creating writable volumes for problematic end-users, we managed to open a VMware GSS case handled by Karan Ahuja(Very helpful support engineer), which ended been worked by the engineering team(Art Rothstein – Champ in AV Eng Team). Note quite alot of logs, memory dumps, and Procmon were exchanged from the problematic VM using various remote gathering techniques. Finally, the fix was determined as a writable volume snapvol.cfg exclusion. (In our case, the problem is caused by smss.exe using a copy of winlogon.exe that is on the writable volume). After putting this exclusion into all problematic end-users, they stopped seeing Black screen issues upon re-connect.

exclude_path=%SystemRoot%\System32\winlogon.exe
Path exclusion in writable volumes snapvol.cfg

In this blog, I am not outlining the steps on how to add the snapvol.cfg exclusion as my ex-colleague Daniel Bakshi outlines on a VMware blog post on how to do it step by step. I hope you will find this information useful if you encounter intermittent black screen issues.

Thanks,
Aresh Sarkari

Create a Memory Dump from a Suspended Virtual Machine – VMware vSAN

10 Nov

If you have a VMware VSAN environment and you wanted to capture a memory dump of the Virtual Machine for debugging or want to provide memory.dmp to VMware GSS or R&D for further analysis go ahead and read further!

Use Case – In our scenario had a few VDI Desktops running Windows 10 1607 + Horizon 7.3.1 + App Volumes Writable Volumes 2.13.1 + UEM 9.2.1 that were getting into unresponsive state. As a last resort we wanted to capture the memory dump to find out more what is causing the VM to get unresponsive.

Step by Step Instructions:

Using the vCenter console select the Virtual Machine VM – Power – Suspend

This will create the *.vmss and *.vmem file for Debugging. (Note the *.vmem file is applicable for ESXi 6.0 onwards)
VM Directory

Make a note of the ESXi host Name/IP for the VM is in Suspend state

— SSH to the ESXi Host and browser to the VM Directory location:

# cd /vmfs/volumes/vsanDatastore/od-av-troub-1 (Where “od-av-troub-1” is the VM name)


— Now lets open the *.vmem file using “cat” command to retrieve the Object ID information. Make a note of the ObjectID

# cat od-av-trou-1-7622414e.vmem

Object ID

In my scenario the Object ID was properly pre-created I didn’t have to use the objtool to find out the Object opened. However, in some cases you might have to run the following command

Now using WINSCP login to the same ESXi Host and go the path:
Object ID – /vmfs/device/vsan/2c86055a-573b-d20a-5cdf-ecf4bbea1e48 (my scenario)
Or/else Object opened at path and download the file “2c86055a-573b-d20a-5cdf-ecf4bbea1e48” which is your ”*.vmem file and move the files to local or remote location that you are using the WINSCP tool.

Rename the Object ID to a friendly name shown in the VM Directory Folder. I renamed it (od-av-trou-1-7622414e.vmem)

For the *.vmss (od-av-trou-1-7622414e.vms) you can directly WINSCP to the ESXi Host and go to the location in the table and move the files to your local or remote location

Once you have both the files *.vmem and *.vmss you can use a VMware Vmss2core Fling and convert it to a dump. Please make sure you meet the requirements and use the appropriate switches to your environment

# vmss2core -W8 od-av-trou-1-7622414e.vmss od-av-trou-1-7622414e.vmem 

— The above command will generate a memory.dmp file which can used in WINDBG for further analysis. If you are sending the dump file to someone make sure use *.zip and compress it before sending.

I hope you will find these steps useful and save a lot of time during daunting unresponsive VM issues. A big thanks to Frank EscarosBuechsel to helping with the entire procedure.

Thanks,
Aresh Sarkari

Troubleshooting Horizon TrueSSO aka Horizon Enrollment Server like a Ninja!

10 Oct

If you have deployed Horizon TrueSSO feature within your environment. Then the most obvious question is how do you troubleshoot during issues? Let me give you some tips and tricks around troubleshooting TrueSSO aka Enrollment Server feature:

  • If you have two teams split one team managing the Active Directory/Certificate Services and other team managing Horizon infrastructure. Then following are the tips for the Horizon Admins. Install the Microsoft RSAT tools on your domain joined machine or Enrollment Servers and install the AD Certificate Services Tools. This will provide you the ability to see the following snap-ins in read-only mode:
    • Enterprise PKI – Allows you to check the CDP and CRL and Issuing CA Status
    • Certificate Templates – TrueSSO, Enrollment Agent (Computer) Templates etc.
SNAGHTML6730c9ff

  • Make sure to Enable the Trace logging on the Enrollment Servers and Horizon Agent (within master image) during troubleshooting. It will provide additional details on the error message
    • [HKEY_LOCAL_MACHINE\SOFTWARE\VMware, Inc.\VMware VDM]
      “debugEnabled”=”true”
      “traceEnabled”=”true”
    • How to know whether the end-users logged in via TrueSSO – Interactive_SmartCard_Logon will be visible in the Horizon Agent (if Trace Log is enable)image
    • If TrueSSO is not used and SAML – CLEAR(Text)_PASSWORD is used you will receive the following in Horizon Agent logs (if trace is enable
      image
  • If you have two Issuing CA’s for High Availability and redundancy then make sure you import the TrueSSO template by Clicking Certificate Templates > New > Certificate Template to Issue. Select “TrueSsoTemplate” from the “Enable Certificate Templates” dialog and press “OK.” on the other Issuing CA. If you skip this step it will complain in Horizon Administrator dashboard – The primary and secondary enrollment server is not connected to the certificate servers “XXXXXX
  • Read and learn to use the VMWare Fling es_diag.exe it will provide a lot of information from the Horizon Enrollment Server stand point and equip you to troubleshoot issues with Certificate Servers.
    • /ListConfigs
    • /ListEnvironment
    • /EnrollmentTest

My colleague Tarique Chowdhury has posted few troubleshooting steps in the following post under Section – Testing it will provide more details as to what to look in the logs.

Log Entries 1
Log Entries 2

I hope you find this post useful during the Horizon TrueSSO aka Enrollment Server troubleshooting.

Thanks,
Aresh Sarkari

Top 10 lessons during Horizon TrueSSO deployment aka Horizon Enrollment Servers

6 Oct

Recently got an opportunity to deploy the VMware Horizon TrueSSO within our environment. TrueSSO provides user with the True SSO (single sign-on) feature, after users log in to VMware Identity Manager (WorkSpaceOne) using a RSA SecurID authentication(optional), users are not required to enter Active Directory credentials in order to use virtual desktop or hosted application.

Let me share my top 10 lessons learnt from the deployment:

  1. In the production deployment recommend to size the Enrollment Server Windows VM as same as the Connection Server(ES role is not very resource intensive)
    • CPU – 4 vCPU
    • Memory – 10 GB RAM
    • HDD – 80 GB
  2. Make sure the “Group Scope” is selected as “Universal” for the  Active Directory Group in which the Enrollment Server – Computer Account is added
  3. On the newly created TrueSSO template (SmartCard Login and Client Authentication) make sure under the Security Tab “Authenticated Users” group has Read permissions and The Active Directory group for the Enrollment Servers (Computer Account) has Read and Enroll
  4. If you are deploying more than one Enrollment Server go in the Horizon ADAM database and add the following value to load balance between two Enrollment Servers:
    cs-view-certsso-enable-es-loadbalance=true
  5. For Large scale AD deployments, it is recommend to add the registry for “ConnectToDomains”=domainname.com
    HKEY_LOCAL_MACHINE\SOFTWARE\VMware, Inc.\VMware VDM\Enrollment Service

    ConnectToDomain
  6. Make Sure the template to be used for TrueSSO, you have selected the check box “Do not store certificate and request in the CA database” and run the following command on the CA server. (without quotes)
    “certutil –setreg DBFlags +DBFLAGS_ENABLEVOLATILEREQUESTS”

    TrueSSO Template Properties
  7. To support Smartcard Logon the following Requirements must be met by the Domain Controller or Kerberos Authentication Certificate:
    • Template name should be Domain Controller or Kerberos Authentication Certificate
      Kerberos Template Properties
    • DNS Name should be selected under Subject Name
      Subject Name Properties
    • Key Usage Extension should be “Digital Signature” and “Key Enciphement
      Key Usage Extension
  8. Make sure the the CA issuing Domain Controller Certificates has the following requirements met (Use GPO’s to deploy the below)
    • Add the Root Certificate to the Enterprise NTAuth Store
    • Add the Root Certificate to Trusted Root Certification Authorities
    • Add an Intermediate Certificate to Intermediate Certification Authorities
  9. Use the True SSO Diagnostic Utility Fling to troubleshoot Enrollment Server, Active Directory PKI Settings and Enterprise CA
  10. On the Domain Controllers under the registry location HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\EnterpriseCertificates\NTAuth\Certificates
    A key with the “Issuing CA Certificate” thumbprint needs to be created on all the domain controllers participating in the TrueSSO. Ideally if the Step 7&8 are done correctly you should not run into this problem. (In our case we had to open-up a Microsoft Case to get this resolved as we were receiving KDC errors.)

My colleague Tarique Chowdhury has written three awesome blog post on the TrueSSO feature make sure to check them out:

Introduction https://blogs.vmware.com/euc/2016/03/true-sso-single-sign-on-view-identity-manager-authenticate.html

Advance https://blogs.vmware.com/euc/2017/02/horizon-7-sso-advanced-features.html

Setting up in Labhttps://blogs.vmware.com/euc/2016/04/true-sso-setting-up-in-a-lab.html

I hope you find this post useful during the Horizon TrueSSO deployment

Thanks,
Aresh Sarkari

Vulnerability Scanner for WannaCry and NoPetya – VDI environments

31 Jul

With a lot of enterprises in the middle of the WannaCry and NoPetya vulnerability. If you are running a enterprise VDI environment the fix is pretty simple. Just target your Master VM or Golden Master images and run the Windows Update. Once you have updated the image simply Recompose or Push-Image the desktops pools with the latest updates. Your environment is quickly secured! These vulnerability reiterate the importance of regular patching within the production environments for your Core infrastructure + Master Images.

WannaCry Patch for All Windows versionshttps://technet.microsoft.com/en-us/library/security/ms17-010.aspx

Vulnerability Scanner

A quick and easy way to scan your environment is using a free EternalBlue vulnerability scanner. – http://omerez.com/eternalblues/

image

Simply download the scanner and launch it on a Windows VM of your choice on Windows 7/8.1/10.

IP Range:
The tool by default tends to select the /24 subnet. However, if you have a bigger subnet like a /19 to scan simply enter the Start and End of the entire subnet range. In this example its a 192.168.0.0/19. It will scan for 8190 IP addresses.

image

I hope you scan your environment ASAP! Get rid of the vulnerability ASAP!

Thanks,
Aresh

Missing default Windows ADMX Templates after importing VMware UEM ADMX files

5 Jul

In VMware User Environment Manager 9.0 (UEM) after you have copied over the VMware UEM Manager GPO’s (.ADMX and .ADML) to the central store for group policy administrative policy templates on a domain controller you cannot view the default Windows ADMX templates such System, Network, Control Panel etc.

Issue
After copying the UEM GPO templates to  \\FQDN\SYSVOL\FQDN\policies\PolicyDefinitions. You cannot see “System” under the Computer Configuration – Policies – Administrative Templates.

What is a Central Store on Domain Controller?
It’s a location to centrally store the .ADMX and .ADML files in a domain environment. The path is as follows:

.ADMX – \\FQDN\SYSVOL\FQDN\policies\PolicyDefinitions
.ADML – \\FQDN\SYSVOL\FQDN\policies\PolicyDefinitions\en-US

MS Reference KB – https://support.microsoft.com/en-in/kb/3087759

Where is the default group policy administrative templates stored?
When central store is not enable the .ADMX and .ADML is stored at the default location on a domain controller. The path is as follows:

.ADMX – C:\Windows\PolicyDefinitions
.ADML – C:\Windows\PolicyDefinitions\en-US

Solution
If you cannot see the Windows default templates post enabling the central store you will have to copy all the ADMX and ADML manually from the Windows default location to Central Store on a domain controller

Copy all the .ADMX/.ADML files from Default to Central Store:

Particulars

Source

Destination

.ADMXC:\Windows\PolicyDefinitions \\FQDN\SYSVOL\FQDN\policies\PolicyDefinitions
.ADMLC:\Windows\PolicyDefinitions\en-US\\FQDN\SYSVOL\FQDN\policies\PolicyDefinitions\en-US

I hope the above steps will help you to get your default Windows ADMX templates back and help you complete the remaining VMware UEM 9.0 server configuration.

Thanks,
Aresh