When Co-Management Goes Bad: The case of Windows 10 IPU and the missing MDM certificate

Windows 10 in-place upgrades are something that many organisations have refined over the years, moving from build to build as to align their machines with support from Microsoft and get the latest feature set of the new OS. Over the past year the same organisations have probably started to enable co-management of devices, to leverage the Endpoint Manager portal to its full extent, or to shift workloads to Intune.

So what happens if a bug breaks your co-management state? In this post we are going to cover this scenario as some people have found themselves having to deal with this issue due to a recent bug.

Windows 10 IPU Certificate Bug

Recently, Microsoft reported issues with any build of Windows 10 from 1809 and higher, running an in place upgrade where an unfortunate bug resulted in the removal of certificates from the certificate store. Below are the full details from Microsoft on the bug as posted here –Windows 10, version 1903 and Windows Server, version 1903 | Microsoft Docs

Certificates may not be present after updating to a newer version of Windows 10

System and user certificates might be lost when updating a device from Windows 10, version 1809 or later to a later version of Windows 10. Devices will only be impacted if they have already installed any Latest cumulative update (LCU) released September 16, 2020 or later and then proceed to update to a later version of Windows 10 from media or an installation source which does not have an LCU released October 13, 2020 or later integrated. This primarily happens when managed devices are updated using outdated bundles or media through an update management tool such as Windows Server Update Services (WSUS) or Microsoft Endpoint Configuration Manager. 

This might also happen when using outdated physical media or ISO images that do not have the latest updates integrated.  

Note Devices using Windows Update for Business or that connect directly to Windows Update are not impacted. Any device connecting to Windows Update should always receive the latest versions of the feature update, including the latest LCU, without any extra steps.  

In-Place Upgrade.. No Problem.. All Appears Well.. Oh Wait..

In the case of co-management, having your certificates scrubbed is a REALLY bad thing due to the fact that machines require an MDM device certificate to communicate with Intune. This process happens during the on-boarding / enrollment process automatically, an example of what you should see from a certificate perspective is below;

So lets paint a picture where you upgrade several thousand devices, after the October updates were applied and just before you serviced the WIM, only to then discover this;

In this case I was first made aware of this issue by a customer, whereby the network team reported a flood of traffic hitting their firewalls going to IP addresses 13.82.96.212 and 104.40.82.191, causing their firewall CPU’s to max out at 100% utilization and in general give everyone a really really bad day getting in our out of their network.

The network team had traced these IP addresses to be part of Microsoft Endpoint Manager IP address range, as listed here – Network endpoints for Microsoft Intune | Microsoft Docs, and had no choice but to block communications to the IP’s in order to save their network.

Looking at the Intune Management Extension logs we can clearly see the issue;

Mass In-Place Upgrade Issues

So if you have had the unfortunate event where you too have upgraded and have this issue, what are your options?

• Microsoft’s Recommendation
  
  Workaround: If you have already encountered this issue on your device, you can mitigate it within the uninstall window by going back to your previous version of Windows using the instructions here. The uninstall window might be 10 or 30 days depending on the configuration of your environment and the version you’re updating to. You will then need to update to the later version of Windows 10 after the issue is resolved in your environment. 
  
  Note Within the uninstall window, you can increase the number of days you have to go back to your previous version of Windows 10 by using the DISM command /Set-OSUninstallWindow. You must make this change before the default uninstall window has lapsed. For more information, see DISM operating system uninstall command-line options. 
  
• MS Support Recommendation
  
  Workaround: Search in the Windows Old folder the Certificates on system context (not user) – C:\Windows\System32\config\systemprofile\AppData\Roaming\Microsoft\SystemCertificates – (here you may have multiple other folders). Manual move the folders with the certificates to the new Windows folder under the exact same location as listed on the Old Windows folder.. Restart the device – Make sure it has DC connectivity and Internet connectivity

But what happens if you have passed the roll back window, cant roll back due to operational reasons, or cant find the certs in the Windows.old?..

There Is Another Way

Fortunately there is a way to force the re-enrollment process, having initially seen a method on another blog (https://www.maximerastello.com/) which appears to have since gone offline.

I manually tested the process of doing the following;

Step 1: Delete The Device
• Delete the device from the Endpoint Manager Portal

Step 2: Delete Scheduled Task
Follow this procedure:
• Run the Task Scheduler as an administrator.
• Go to Task Scheduler Library > Microsoft > Windows > EnterpriseMgmt. Write down the enrollment ID somewhere, you will need it for the cleanup.
• Delete all the existing tasks the enrollment folder.
• Delete the enrollment ID folder.

Step 3: Registry Key Deletion
Use the previous enrollment ID to search the registry:
• Open the Registry Editor as an administrator.

Search for the enrollment ID you wrote in the following locations and if found, delete the key that is containing the ID:
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Enrollments\xxxxxxxxxxxxx
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Enrollments\Status\xxxxxxxxxxxxx
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\EnterpriseResourceManager\Tracked\xxxxxxxxxxxxx
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\PolicyManager\AdmxInstalled\xxxxxxxxxxxxx
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\PolicyManager\Providers\xxxxxxxxxxxxx
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Provisioning\OMADM\Accounts\xxxxxxxxxxxxx
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Provisioning\OMADM\Logger\xxxxxxxxxxxxx
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Provisioning\OMADM\Sessions\xxxxxxxxxxxxx
DO NOT delete registry keys that are not in the list above. They will be overwritten after the new enrollment.

Step 4: Force Reenrollment
To be properly executed, the enrollment command must be entered in a SYSTEM context.

• Use PSExec to launch a Command Prompt as SYSTEM: psexec /i /s cmd
• In the Command Prompt, enter the following command: c:\windows\system32\deviceenroller.exe /c /AutoEnrollMDM

In this case, the process worked, but be patient for the device to update in the portal;

PowerShell Automation

Taking the manual process and constructing it into PowerShell allows you to deploy a script to the affected devices. So here it is, a script that you can deploy either via a CB, first of all checking for machines without an Intune MDM cert, or as a script to push out to your devices.

Note: You will need to bulk delete your affected devices in Intune first, which can be performed via Bulk Actions after filtering the machine list based on the last check in date, or obtaining a device list by using a CI to check devices for the MDM certificate presence.

Note that deleting your devices from Endpoint Manager and running this script is provided without warranty, do so at your own risk.

<#	
	.NOTES
	===========================================================================
	 Created on:   	06/11/2020 14:23
	 Created by:   	MauriceDaly
	 Filename:     	Invoke-MDMReenrollment.ps1
	===========================================================================
	.DESCRIPTION
		Force re-enrollment of Intune managed devices due to Intune MDM certificate
		issues during IPU
#>

#region Functions

# Logging Function
function Write-LogEntry {
	param (
		[parameter(Mandatory = $true, HelpMessage = "Value added to the log file.")]
		[ValidateNotNullOrEmpty()]
		[string]$Value,
		[parameter(Mandatory = $true, HelpMessage = "Severity for the log entry. 1 for Informational, 2 for Warning and 3 for Error.")]
		[ValidateNotNullOrEmpty()]
		[ValidateSet("1", "2", "3")]
		[string]$Severity,
		[parameter(Mandatory = $false, HelpMessage = "Name of the log file that the entry will written to.")]
		[ValidateNotNullOrEmpty()]
		[string]$FileName = "Invoke-MDMReenrollment.log"
	)
	# Determine log file location
	$global:LogFilePath = Join-Path -Path $Env:windir -ChildPath "CCM\Logs\$FileName"
	
	# Construct time stamp for log entry
	$Time = -join @((Get-Date -Format "HH:mm:ss.fff"), " ", (Get-WmiObject -Class Win32_TimeZone | Select-Object -ExpandProperty Bias))
	
	# Construct date for log entry
	$Date = (Get-Date -Format "MM-dd-yyyy")
	
	# Construct context for log entry
	$Context = $([System.Security.Principal.WindowsIdentity]::GetCurrent().Name)
	
	# Construct final log entry
	$LogText = "<![LOG[$($Value)]LOG]!><time=""$($Time)"" date=""$($Date)"" component=""Invoke-MDMReenrollment"" context=""$($Context)"" type=""$($Severity)"" thread=""$($PID)"" file="""">"
	
	# Add value to log file
	try {
		Out-File -InputObject $LogText -Append -NoClobber -Encoding Default -FilePath $global:LogFilePath -ErrorAction Stop
	} catch [System.Exception] {
		Write-Warning -Message "Unable to append log entry to Invoke-MDMReenrollment.log file. Error message: $($_.Exception.Message)"
	}
}
#endregion Functions

#region Variables

$RegistryKeys = "HKLM:\SOFTWARE\Microsoft\Enrollments", "HKLM:\SOFTWARE\Microsoft\Enrollments\Status","HKLM:\SOFTWARE\Microsoft\EnterpriseResourceManager\Tracked", "HKLM:\SOFTWARE\Microsoft\PolicyManager\AdmxInstalled", "HKLM:\SOFTWARE\Microsoft\PolicyManager\Providers","HKLM:\SOFTWARE\Microsoft\Provisioning\OMADM\Accounts", "HKLM:\SOFTWARE\Microsoft\Provisioning\OMADM\Logger", "HKLM:\SOFTWARE\Microsoft\Provisioning\OMADM\Sessions"

#endregion Variables

try {
	Write-LogEntry -Value "[MDM Enrollment Issue Detection Script] Checking for MDM certificate in computer certificate store" -Severity 1
	
	# Check for MDM device certificate
	$IntuneMDMCert = Get-ChildItem -Path Cert:\LocalMachine\My | Where-Object {
		$_.Issuer -match "Intune MDM"
	}
	
	if ($IntuneMDMCert.Count -eq 0) {
		Write-LogEntry -Value "[MDM Issue Detected] Intune MDM certificate does not exist. Starting forced on-boarding process." -Severity 1
		
		# Obtain current management GUID from Task Scheduler
		$EnrollmentGUID = Get-ScheduledTask | Where-Object {$_.TaskPath -like "*Microsoft*Windows*EnterpriseMgmt*"} | Select-Object -ExpandProperty TaskPath -Unique | Where-Object {$_ -like "*-*-*"} | Split-Path -Leaf

		# Start clean up process
		if (-not [string]::IsNullOrEmpty($EnrollmentGUID)) {
			Write-LogEntry -Value "- Current enrollment GUID detected as $([string]$EnrollmentGUID)" -Severity 1
			
			# Stop Intune Management Exention Agent and CCM Agent services
			Write-LogEntry -Value "- Stopping MDM services" -Severity 1
			if (Get-Service -Name IntuneManagementExtension) {
				Write-LogEntry -Value "- Stopping IntuneManagementExtension service..." -Severity 1
				Stop-Service -Name IntuneManagementExtension
			}
			Write-LogEntry -Value "- Stopping CCMExec service..." -Severity 1
			Stop-Service -Name CCMExec
			
			# Remove task scheduler entries
			Write-LogEntry -Value "- Removing task scheduler Enterprise Management entries for GUID - $([string]$EnrollmentGUID)" -Severity 1
			Get-ScheduledTask | Where-Object {$_.Taskpath -match $EnrollmentGUID} | Unregister-ScheduledTask -Confirm:$false
			
			foreach ($Key in $RegistryKeys) {
				Write-LogEntry -Value "- Processing registry key $Key" -Severity 1
				# Remove registry entries
				if (Test-Path -Path $Key) {
					# Search for and remove keys with matching GUID
					Write-LogEntry -Value "- GUID entry found against key $Key. Removing.." -Severity 1
					Get-ChildItem -Path $Key | Where-Object {$_.Name -match $EnrollmentGUID} | Remove-Item -Recurse -Force -Confirm:$false -ErrorAction SilentlyContinue
				}
			}
			
			# Start Intune Management Exention Agent service
			Write-LogEntry -Value "- Starting MDM services" -Severity 1
			if (Get-Service -Name IntuneManagementExtension) {
				Write-LogEntry -Value "- Starting IntuneManagementExtension service..." -Severity 1
				Start-Service -Name IntuneManagementExtension
			}
			Write-LogEntry -Value "- Starting CCMExec service..." -Severity 1
			Start-Service -Name CCMExec
			
			# Sleep
			Write-LogEntry -Value "- Waiting for 30 seconds prior to running DeviceEnroller" -Severity 1
			Start-Sleep -Seconds 30
			
			# Start re-enrollment process
			Write-LogEntry -Value "- Calling C:\Windows\System32\DeviceEnroller.exe with /C /AutoenrollMDM switches" -Severity 1
			$EnrollmentProcess = Start-Process -FilePath "C:\Windows\System32\DeviceEnroller.exe" -ArgumentList "/C /AutoenrollMDM" -NoNewWindow -Wait -PassThru
			
		} else {
			Write-LogEntry -Value "Unable to obtain enrollment GUID value from task scheduler. Aborting" -Severity 3; exit 1
		}
	} else {
		Write-LogEntry -value "Device machine certifice exists. Aborting" -Severity 2; exit 1
	}
	
} catch [System.Exception] {
	Write-LogEntry -value "Error message: $($_.Exception.Message)" -Severity 3; exit 1
}

The Results

Always test on a small subset of devices first, but the results should be something like below;

Devices have re-appeared, are co-managed once again and the check in date should be up to date.

Conclusion

This method allows for automated re-onboarding of devices which are in a co-managed state and have been impacted by this IPU bug. Hotfixes are in the process of being released by Microsoft, and of course, make sure you are always using the latest ISO’s and serviced WIM’s before deployment.

Thanks for reading