Randomize script start time

Any suggestions on how to randomize the start time of a policy scheduled script? The script will need to run for 5-15 minutes on every asset and upload a large file. So it is not ideal to run it on multiple workstations simultaneously as it would cripple their Internet bandwidth and probably timeout thus not finishing the uploads properly. I would like to spread the load over multiple hours or even days. A Syncro feature in the Policy Script Schedule with Daily/Weekly/Monthly and a Time slot of Random instead of a specific hour would be ideal. Another option would be a time range with selectable hours. For example between 8:PM - 5:AM.

Any ideas other than manually scheduling this script per asset?

Not available currently. With the new PI, you can go 5 folders deep, so you could create policies with different start times and segment them off, but beyond that, there is no start window available. Could you do a PS1 and use task scheduler to randomize?

It’s a bit of work but that could do the trick. I will work on and report back here.

You can modify the Script with Get-Random and Start-Sleep.

# Example 
Write-Output "START"

$sleepTime = Get-Random -Minimum 0 -Maximum 10 # change the Maximum to 900 for your 15min
Start-Sleep $sleepTime

Write-Output "WOHOO i Sleeped: $sleepTime"
1 Like

The problem with that is the time allowed for the script to run in Syncro would have to be set to hours which would block anything else from running.

Upstream Internet bandwidth is the reason for randomizing the run time.

the main problem with this @francis.pimenta is that it will hold execution of your script for that random time all the time. If it is a scheduled run, that’s fine, but if you wanted to run manually “oops, I rolled a 9.5 minute wait on this 1 machine” - that sucks.

You could mitigate this by adding a script variable to bypass the wait time

or by letting the user pass a max number of seconds to wait

I haven’t been able to track down the “Max Script Run Time” down from inside the running script, so that just has to manually be long enough for your script to run + your Randomized wait time too

Upstream internet bandwidth is one possible reason, for me it’s API Calls through “Set-AssetField” and such, and each of those commands calls an API Request.
With all of a location running at the same time - it very easily triggers 429 TOO MANY REQUESTS responses

Whoa, how many asset custom fields are you updating at the same time and on that type of frequency? Curious what the use case is there.

We have about 20 custom Asset fields that we are setting, scheduled to run once a day, ranging from various things like

  • whether the device is onboarded in various Agents
    • Huntress
    • Defender ATP
    • Monitoring agent
    • ConnectWise Control agent
    • FSLogix installed/Version
    • Backup agents
  • If unapproved Remote Control software is found (LogMeIn, TeamViewer, CW Control to instances other than ours, etc)
  • more complete listing of other AV Providers that Windows Sees as Active
    • Eg even when we uninstalled WebRoot for Defender ATP, Webroot would often still be registered and this would block Defender ATP from registering fully/automatically
  • If AV Signatures are up to date
  • AV Protection Status (realtime enabled, cloud protection enabled, firewall enabled, etc)
  • Windows activation
  • SSD vs HDD boot drive
  • BIOS Version

Some of these may also raise an alert and/or trigger other automation to run (like installing Huntress, or trying to run a Windows Activation, etc) but the remediations aren’t really a problem, cause they happen later on.

And other areas like an

  • output from quser for the Citrix Hosts, PVS Hosts, MCS Hosts, WVD/AVD Hosts, RDS Hosts.
  • BitLocker drive ID’s / Status.
  • best-effort Listing of Monitors attached - sometimes it can detect size/type/brand of monitors - depending on connectors

We make an effort to spread them out into several various related jobs with various repetition schedules - not just one large script. For example we might check

  • Warranty status once per week,
  • quser run maybe once an hour so it’s pretty fresh,
  • check the Agents/AV are onboarded/active status once per day
    but there is only so much you can do without manually creating three groups for each large customer segment, and assigning one at 10:10am, one at 10:30am, and one at 10:50am or similar.

But the MAIN issue that is really multiplying the problem, is the number of devices behind a single public IP - so it isn’t just 20 Fields being updated, it’s 20 fields * 60 devices, all running at roughly the same few seconds, hitting the API at the same time.

Spreading it out manually works, and spreading it out over a larger period like the randomizing mentioned above works too, BUT it keeps the scripting engine blocked for sometimes 8 minutes. Or worse if it was offline for a while/days/weeks and we had a couple items checked for “run once you are back online” - then you might have 6 or 8 or MORE stacked up all wanting to run immediately after each other

Support’s answer a few months ago was

There is a rate limit of 120 request per minute per IP address. I can submit a feature request for you on any suggestions you have on the API / Scheduled Scripts. A possible work around for the Scheduled Scripts / rate-limit is to clone a RMM policy in 2 or 3 different groups and set the scheduled Scripts to run at different times based on the group.

so if I assuming that calls like Set-Asset-Field, RMM-Alert, and Close-RMM-Alert all take 1 API Call each, one of our larger scripts might be hitting 15 per host with 40+ hosts behind the same public IP.

So easily ~600 requests that would have to be spread out to ~120 / minute = ~5 minutes
Add some extra buffer there for parts that complete faster or slower hitting around the same time, regular background calls from Syncro directly, and any other actions/troubleshooting/remediation we might be working on at the same time, and you could easily need those to be spread out over ~8 minutes+ easily.

Add a few “Offline” scripts that needed to catch up, or trying to push another script down to fix something, and we easily hit the API limit daily or more often and the scripts just error, unless we manually implement additional spreading

Yeah they do consume one call each. I am going to think on this because there has to be a more efficient way to do this. What are you doing with that data on any given day if it’s not being used to track/triage exceptions?

In regard to randomizing the script times, that isn’t something on the radar. So I am trying to think of another way to make this work for you.

Yeah, my suggestions included

  • building multiple custom asset field updates into one query
  • batching the queries some other way
  • Adjusting script-run option to stagger out the hosts start-execution time over the course of 15 minutes or so
  • Detect whether a Script-run is Scheduled or Manually run, so that include a random delay in our script while it is scheduled to stagger when the queries will hit.

I was super interested in that last option, but the current way the scripts run are basically Powershell.exe -Command "& C:\path\GUID.ps1" and some additional stuff for the variables inline, from one of the Syncro EXE’s with some extra options, but nothing that I could identify as Scheduled versus On-Demand runtime - unless I manually created and “IsOnDemand” parameter.

One thing I would HIGHLY suggest to Syncro is just creating a default, read only variable with Context Information, similar to how the normal Platform variables can be used, but just automatic and read-only

Could include some useful info like (hypothetical variable)
$SyncroContext.CustomerName > “Customer One”
$SyncroContext.DeviceID > Syncro DeviceGUID
$SyncroContext.PolicyName = “WorkStation-UpdateNightly”
$SyncroContext.MaxExecutionTime = [timespan]::FromSeconds(600) = 10 minutes
$SyncroContext.ScriptName = “Check MS Defender Onboarded”
SyncroContext.RunType = “Scheduled” or “On-demand”
etc, and more possibly, like… maybe “Number of devices this is scheduled to run on now” ?
Which could be made to include RunningScript or ScheduledExecutionTime or IsPastDue (like Azure Functions provides if a scheduled run was not able to happen at the scheduled time)

Anyway, if there was something like that, you could at least differentiate between scheduled runs, vs on-demand so you could decide to either stagger over X minutes, or run immediately, etc without having to manually create loads of “Platform” variables or enter it every time.

cough Syncro could also just implement staggered API Backoffs in their PowerShell module, staggering the start time when it’s starting on bulk amount of devices, and specialized API Limits/treatments for the authentication the PowerShell Module uses cough

Sorry, I meant a more efficient way for you to do it than updating so many custom fields on a regular basis that it’s enforcing throttling. For instance, just triggering updates when exceptions are met versus passing checks.

At any rate, I’ll keep a lookout to see if this becomes a popular request. To date, we haven’t had many requests for this.

Oh I see what Andy is saying. Add an if statement to check each of these fields before updating them. If there is no change, don’t update the field. That will significantly reduce your API calls.

Yes, this is definitely part of it. Send the current values of the variables down with the script, and only update them if they have changed.

Ah, right. You could add all of the fields you want to potentially update to the script as a Script variable - type platform - and pass a reference down to the script.

Check it it changed, and update if needed.

But I would not be surprised if that just essentially does a get item from the API anyway, so it would be the same number of API calls, or more.

So let’s say there are 15 items you were going to check / update if needed

The way we’re doing it now takes 15 API calls to set 15 items
If it had to fetch them beforehand and then set them if needed, I wouldn’t be surprised if it started off with 15 API calls to fetch them all, and then used six calls to write back the six that it changed.

Although… Fetching them all down the first time could be bunched up into one probably

But those fetches are inside Syncro in the pre-script setup before it’s passed to the client. So they probably don’t count. It’s the external updates from the clients inbound that count.

You could use PowerShell to create a one-time Windows Scheduler Event with randomized start time.

None of this would be an issues if Syncro had proper search/filter based policies/rules like the big 4 RMMs. Yes I’m still bitter they revamped policies and only went halfway. Anyway, one solution to this would be to cache the custom fields locally in file(s) or registry key(s). Script checks current values against the values last time the script ran. If there’s a difference, then it makes the API call. The only issue with this might be if a value gets changed on Syncro’s side for some reason, it will never get overwritten by the current status until the status changes. Not sure that would really be an issue but if you wanted you could build in a once in every x runs or something it actually resends all the values to Syncro.

I have just noticed that I am getting hit very hard by this “too many requests” errors, when I was tracking down some inaccurate results. I have lots of custom fields generating custom named reports that are not possible with Syncro’s report generator. So sign me up as another customer requesting some kind of script time randomizer, or increase the number of requests we can have, or change your API to where it can update more than one custom field for each call.