Scripts with API functions are showing up as successes even if the API call fails

TL;DR: Even when scripts with API functions like Set-AssetField run into web exceptions due to rate limiting, the scripts are showing in history as successes.

We have several scripts to perform functions like checking for AV solutions, checking disk space and resource use, checking for needed reboots, etc. Many of these scripts are writing back to the API so we can then run reports and searches based on the custom fields we’re setting. A problem I discovered yesterday is that many of our calls to write back to custom asset fields have been failing without our knowledge. I only discovered this because there were assets showing up in searches they shouldn’t have been. I looked deeper and realized that a lot of fields were blank or outdated.

Looking at the script history I saw nothing but successes, however when I looked at the details on many of them I saw this popping up all the time:

error> System.Net.WebException: The remote server returned an error: (429) Too Many Requests.
error> HTTP Status: 429 '429'
error> Content-Type: ''
error> Stack:
error>   at Invoke-WebRequest20, C:\ProgramData\Syncro\bin\module.psm1: line 341
error>   at Call-Api, C:\ProgramData\Syncro\bin\module.psm1: line 268
error>   at Call-SyncroApi, C:\ProgramData\Syncro\bin\module.psm1: line 249
error>   at Set-Asset-Field, C:\ProgramData\Syncro\bin\module.psm1: line 150
error>   at <ScriptBlock>, C:\ProgramData\Syncro\bin\afe37143-4e00-4213-b557-7c405ec29b32.ps1: line 67
error>   at <ScriptBlock>, <No file>: line 1
error>   at <ScriptBlock>, <No file>: line 1
error> Call-SyncroApi: failure

I’ve significantly reduced the frequency of this error on some scripts by pulling in custom fields as variables and comparing the values before attempting unnecessary API writes, but some of the fields we’re updating are real-time data that is - in some cases - critical and would cause issues if it was incorrect when we needed it. I attempted to resolve this with a try/catch block, however it turns out that the way the Syncro module is handling System.Net.WebExceptions doesn’t trigger a try/catch block because it’s not throwing an exception at the script. It just writes the data to $error and displays it.

What I’ve ended up doing is using a do/while loop that introduces a random wait between 10 and 120 seconds, clears $error, attempts the API writes, converts the contents of $error to a string and checks it for “429”, then restarts the process until $error no longer contains that string.

Update: I realized my first fix wasn’t going to work for one of my scripts because of how unwieldy it would be. I wrote this function to wrap around the Set-Asset-Field function instead.

# This function wraps around the Syncro Set-Asset-Field function, attempting to avoid API rate limits.
# It will first compare the current value (which should be pulled from a Syncro Platform Variable using the field to be updated), and stop if the values match.
# It will then clear the system $error variable, then attempt to set the asset field.  If successful, it logs a message and exits.
# Success in this case means the Syncro module didn't write a "429" error to the $error field.
# If the rate limit prevents the update, the function waits a random time from 10-60 seconds before clearing $error and retrying.
# Function may be used in two ways:
# UpdField "Field Name" "Current Value" "New Value"
# UpdField -fn "Field Name" -cv "Current Value" -nv "New Value"
# If used without parameter names, parameters must be in the specified order.
# New Value may be a string or variable, but variable type must be string.
function UpdField {
	Param
		(
			[Alias("fn")]
			[Parameter(Mandatory=$true, Position=0)]
			[string] $FieldName,
			[Alias("cv")]
			[Parameter(Mandatory=$true, Position=1)]
			[string] $CurrVal,
			[Alias("nv")]
			[Parameter(Mandatory=$true, Position=2)]
			[string] $NewVal
		)
	$RandSec = 0
	$UpdAttempts = 1
	write-host ("Attempting to update Syncro field `"" + $FieldName + "`" with value `"" + $NewVal + "`", from current value `"" + $CurrVal + "`".")
	write-host "----------"
	if($CurrVal -ne $NewVal){
		do{
			$error.clear()
			Start-Sleep -Seconds $RandSec
			write-host ("Update attempt " + $UpdAttempts)
			Set-Asset-Field -Name $FieldName -Value $NewVal
			$errorstr = $error | out-string
			$errorval = $errorstr.Contains("429")
			if($errorval -eq $False){
				write-host ("Successfully updated Syncro field `"" + $FieldName + "`" with value `"" + $NewVal + "`".")
			}else{
				$RandSec = Get-Random -Minimum 10 -Maximum 60
				write-host ("Failed to write to API due to rate limit.  Sleeping for " + $RandSec + " seconds.")
				write-host "----------"
				$UpdAttempts++
			}
		}while($errorval -eq $True)
	}else{
		write-host "Current and new values for field match.  Skipping."
		write-host "----------"
	}
}