Any suggestions on how to randomize the start time of a policy scheduled script? The script will need to run for 5-15 minutes on every asset and upload a large file. So it is not ideal to run it on multiple workstations simultaneously as it would cripple their Internet bandwidth and probably timeout thus not finishing the uploads properly. I would like to spread the load over multiple hours or even days. A Syncro feature in the Policy Script Schedule with Daily/Weekly/Monthly and a Time slot of Random instead of a specific hour would be ideal. Another option would be a time range with selectable hours. For example between 8:PM - 5:AM.
Any ideas other than manually scheduling this script per asset?
Not available currently. With the new PI, you can go 5 folders deep, so you could create policies with different start times and segment them off, but beyond that, there is no start window available. Could you do a PS1 and use task scheduler to randomize?
You can modify the Script with Get-Random and Start-Sleep.
# Example
Write-Output "START"
$sleepTime = Get-Random -Minimum 0 -Maximum 10 # change the Maximum to 900 for your 15min
Start-Sleep $sleepTime
Write-Output "WOHOO i Sleeped: $sleepTime"
the main problem with this @francis.pimenta is that it will hold execution of your script for that random time all the time. If it is a scheduled run, that’s fine, but if you wanted to run manually “oops, I rolled a 9.5 minute wait on this 1 machine” - that sucks.
You could mitigate this by adding a script variable to bypass the wait time
I haven’t been able to track down the “Max Script Run Time” down from inside the running script, so that just has to manually be long enough for your script to run + your Randomized wait time too
Upstream internet bandwidth is one possible reason, for me it’s API Calls through “Set-AssetField” and such, and each of those commands calls an API Request.
With all of a location running at the same time - it very easily triggers 429 TOO MANY REQUESTS responses
We have about 20 custom Asset fields that we are setting, scheduled to run once a day, ranging from various things like
whether the device is onboarded in various Agents
Huntress
Defender ATP
Monitoring agent
ConnectWise Control agent
FSLogix installed/Version
Backup agents
If unapproved Remote Control software is found (LogMeIn, TeamViewer, CW Control to instances other than ours, etc)
more complete listing of other AV Providers that Windows Sees as Active
Eg even when we uninstalled WebRoot for Defender ATP, Webroot would often still be registered and this would block Defender ATP from registering fully/automatically
If AV Signatures are up to date
AV Protection Status (realtime enabled, cloud protection enabled, firewall enabled, etc)
Windows activation
SSD vs HDD boot drive
BIOS Version
Some of these may also raise an alert and/or trigger other automation to run (like installing Huntress, or trying to run a Windows Activation, etc) but the remediations aren’t really a problem, cause they happen later on.
And other areas like an
output from quser for the Citrix Hosts, PVS Hosts, MCS Hosts, WVD/AVD Hosts, RDS Hosts.
BitLocker drive ID’s / Status.
best-effort Listing of Monitors attached - sometimes it can detect size/type/brand of monitors - depending on connectors
We make an effort to spread them out into several various related jobs with various repetition schedules - not just one large script. For example we might check
Warranty status once per week,
quser run maybe once an hour so it’s pretty fresh,
check the Agents/AV are onboarded/active status once per day
but there is only so much you can do without manually creating three groups for each large customer segment, and assigning one at 10:10am, one at 10:30am, and one at 10:50am or similar.
But the MAIN issue that is really multiplying the problem, is the number of devices behind a single public IP - so it isn’t just 20 Fields being updated, it’s 20 fields * 60 devices, all running at roughly the same few seconds, hitting the API at the same time.
Spreading it out manually works, and spreading it out over a larger period like the randomizing mentioned above works too, BUT it keeps the scripting engine blocked for sometimes 8 minutes. Or worse if it was offline for a while/days/weeks and we had a couple items checked for “run once you are back online” - then you might have 6 or 8 or MORE stacked up all wanting to run immediately after each other
There is a rate limit of 120 request per minute per IP address. I can submit a feature request for you on any suggestions you have on the API / Scheduled Scripts. A possible work around for the Scheduled Scripts / rate-limit is to clone a RMM policy in 2 or 3 different groups and set the scheduled Scripts to run at different times based on the group.
so if I assuming that calls like Set-Asset-Field, RMM-Alert, and Close-RMM-Alert all take 1 API Call each, one of our larger scripts might be hitting 15 per host with 40+ hosts behind the same public IP.
So easily ~600 requests that would have to be spread out to ~120 / minute = ~5 minutes
Add some extra buffer there for parts that complete faster or slower hitting around the same time, regular background calls from Syncro directly, and any other actions/troubleshooting/remediation we might be working on at the same time, and you could easily need those to be spread out over ~8 minutes+ easily.
Add a few “Offline” scripts that needed to catch up, or trying to push another script down to fix something, and we easily hit the API limit daily or more often and the scripts just error, unless we manually implement additional spreading
Yeah they do consume one call each. I am going to think on this because there has to be a more efficient way to do this. What are you doing with that data on any given day if it’s not being used to track/triage exceptions?
In regard to randomizing the script times, that isn’t something on the radar. So I am trying to think of another way to make this work for you.
building multiple custom asset field updates into one query
batching the queries some other way
Adjusting script-run option to stagger out the hosts start-execution time over the course of 15 minutes or so
Detect whether a Script-run is Scheduled or Manually run, so that include a random delay in our script while it is scheduled to stagger when the queries will hit.
I was super interested in that last option, but the current way the scripts run are basically Powershell.exe -Command "& C:\path\GUID.ps1" and some additional stuff for the variables inline, from one of the Syncro EXE’s with some extra options, but nothing that I could identify as Scheduled versus On-Demand runtime - unless I manually created and “IsOnDemand” parameter.
One thing I would HIGHLY suggest to Syncro is just creating a default, read only variable with Context Information, similar to how the normal Platform variables can be used, but just automatic and read-only
Could include some useful info like (hypothetical variable) $SyncroContext.CustomerName > “Customer One” $SyncroContext.DeviceID > Syncro DeviceGUID $SyncroContext.PolicyName = “WorkStation-UpdateNightly” $SyncroContext.MaxExecutionTime = [timespan]::FromSeconds(600) = 10 minutes $SyncroContext.ScriptName = “Check MS Defender Onboarded” SyncroContext.RunType = “Scheduled” or “On-demand”
etc, and more possibly, like… maybe “Number of devices this is scheduled to run on now” ?
Which could be made to include RunningScript or ScheduledExecutionTime or IsPastDue (like Azure Functions provides if a scheduled run was not able to happen at the scheduled time)
Anyway, if there was something like that, you could at least differentiate between scheduled runs, vs on-demand so you could decide to either stagger over X minutes, or run immediately, etc without having to manually create loads of “Platform” variables or enter it every time.
cough Syncro could also just implement staggered API Backoffs in their PowerShell module, staggering the start time when it’s starting on bulk amount of devices, and specialized API Limits/treatments for the authentication the PowerShell Module uses cough
Sorry, I meant a more efficient way for you to do it than updating so many custom fields on a regular basis that it’s enforcing throttling. For instance, just triggering updates when exceptions are met versus passing checks.
At any rate, I’ll keep a lookout to see if this becomes a popular request. To date, we haven’t had many requests for this.
Oh I see what Andy is saying. Add an if statement to check each of these fields before updating them. If there is no change, don’t update the field. That will significantly reduce your API calls.
Ah, right. You could add all of the fields you want to potentially update to the script as a Script variable - type platform - and pass a reference down to the script.
Check it it changed, and update if needed.
But I would not be surprised if that just essentially does a get item from the API anyway, so it would be the same number of API calls, or more.
So let’s say there are 15 items you were going to check / update if needed
The way we’re doing it now takes 15 API calls to set 15 items
If it had to fetch them beforehand and then set them if needed, I wouldn’t be surprised if it started off with 15 API calls to fetch them all, and then used six calls to write back the six that it changed.
Although… Fetching them all down the first time could be bunched up into one probably
But those fetches are inside Syncro in the pre-script setup before it’s passed to the client. So they probably don’t count. It’s the external updates from the clients inbound that count.
None of this would be an issues if Syncro had proper search/filter based policies/rules like the big 4 RMMs. Yes I’m still bitter they revamped policies and only went halfway. Anyway, one solution to this would be to cache the custom fields locally in file(s) or registry key(s). Script checks current values against the values last time the script ran. If there’s a difference, then it makes the API call. The only issue with this might be if a value gets changed on Syncro’s side for some reason, it will never get overwritten by the current status until the status changes. Not sure that would really be an issue but if you wanted you could build in a once in every x runs or something it actually resends all the values to Syncro.
I have just noticed that I am getting hit very hard by this “too many requests” errors, when I was tracking down some inaccurate results. I have lots of custom fields generating custom named reports that are not possible with Syncro’s report generator. So sign me up as another customer requesting some kind of script time randomizer, or increase the number of requests we can have, or change your API to where it can update more than one custom field for each call.
