Supress Alerts with Scheduling

We have servers that run in the public cloud (ie Google Cloud, AWS, Azure) and some of these resources are shut down between 1AM and 5AM for maintenance or simply to save on public cloud infrastructure costs.

Of course when these servers shut down each evening we receive an alert from the Syncro Agent.
We request that there be a feature of alert supression with a schedule so that it can be scheduled each evening between 1AM and 5AM.

Schedule should allow day of the month or every XXth day of the month or on a specific day as options.

Please seriously consider this as most RMM platforms have the ability to supress alerts for standard maintenance cycles on a schedule and I cannot find a way to do this in Syncro today.

Thank You!

2 Likes

You could run a script on a schedule that triggers maintenance mode for 4 hours…

2 Likes

Would be much easier if the UI has a feature for this as when engineers are working on servers on a planned outage they can set maintenance mode manually…but when it is recurring I don’t see a way to schedule this.

Can you provide a script code example on how to accomplish this or provide a platform variable to use to make the script check on maintenance mode for the asset?

Thanks!

Possibly what I should also ask for here is also the ability to schedule things within a finite run time window. For example on Monday October 18th, 2021 at 1AM PDT as the start time much like a Windows Task Scheduler. This way if we know we want to schedule something we can be assured that it will run at a specific time.

Thanks

While it is possible to enable maintenance windows via script, there should be a way to do it 100% server side, as script runs are not 100%, and if your script run is delayed for w/e reason, a maintenance window scheduled ahead of time should take effect at the correct time.

I think this is related enough to have as a note, but there should be a setting so Syncro Scheduled reboots can automatically happen w/o triggering offline alerts. When I run windows updates and tell Syncro to auto reboot, I do not need to be alerted that the system is offline while it installs updates. Its important that alerts can be controlled to only trigger when they are actionable.

Agreed. Updates in progress should not alert that the asset was offline.

Scheduled maintenance mode also should not alert that the asset went offline.

Syncro needs the ability to schedule maintenance mode on a regular schedule if needed as in my case so that the alerts are suppressed. OR the ability to supress alerts on a schedule other than up to 4 hours as is the case now with scripting that only allows a schedule for maintenance mode up to a MAXIMUM of 4 hours. We need the ability to schedule a maintenance mode via scripting for more than 4 hours. Syncro could really benefit from some sort of scheduling / jobs addition to the software so that scripting as it runs becomes a job on a schedule.

It’s a checkbox on scripts to have them enable maintenance mode. You don’t need any code to make it work.

HI Andy,

Thanks for the reply.

Yes we see that but it’s limited to 4 hours per script and we would like to see a full job scheduler so that this can be setup beyond 4 hours on a single job/script.

Thanks,

Richard

Do you regularly schedule maintenance lasting longer than 4 hours? If so, can you please elaborate on the use case(s) behind this? Normally for maintenance lasting that long you’d just trigger the maintenance manually and disable it when the work is complete.

Yes the use case is for public cloud infrastructure. We shut down the servers on our public cloud infrastructure to save on computing costs when the servers are not in use in the middle of the night. We have Syncro agents installed on them and they fire off both alerts and emails when the servers go down intentionally. We wish to be able to suppress these alerts and email alerts on a regular scheduled basis for more than 4 hours as sometimes we wish to shut down the servers for say 10 hours or 6 hours for example when they are not in use. This saves our public cloud spend each month.

Thanks,

Richard Young

image001.png

Got it, thanks. This is a definitely an edge case, and one I haven’t encountered before. Thanks for the additional details.

The quickest way to make notifications useless is to add a lot of noise. At that point they start getting ignored.

Anything that can help reduce the noise is very useful. Here are several time based reasons to filter out alerts.

  • Night time processes that run high CPU and memory and are expected.
  • Night and Weekend internet outages from devices that are unimportant.
  • Night and weekend monitored printers and other devices that are turned off to save power.
  • Backup processes that cause high resource utilization and are expected.
1 Like

Hey Gary, you can do this today through Automated Remediations. Just use the “Business Hours” condition and you can filter whatever you want out so that it doesn’t fire outside of your business hours.

Thanks Andy, That will be useful, however I do not see how I can apply that to alerts like device offline, high cpu, or like ping system checks.

So within the same Automation, check out the condition for trigger category. There you can define what trigger any give remediation would apply to. So if you want to filter one during business hours while leaving others in play, that is totally possible within that framework.

Please fix it. Fixed 4 hours is either too long, or not long enough. We are chasing enough noise already. Example, If the machine hangs on restart after a script is run or restart issued, or some service fails to start, its not proactive to wait for 4 hours for maintenance to be suspended. (Checkbox, place machine in maintenance mode before starting script, Checkbox, disable maintenance mode on restart, or after x hours. As it is, we are either waiting / watching for an auto-remediation, or chasing someone else’s restart / server offline.

Hey @rick.cassel,

I think Andy was using 4 hours as an example because that is the time frame mentioned in the original post. There are a few options available for amount of time the machine stays in maintenance mode.

It is not shown in the screenshot, but 4 hours is the longest option available.

Thanks! That’s helpful. For regular reboots, 30 minutes will work. - RC

Yeah you have a bunch of options with 4 hours being the max.

The problem is that there is not real job scheduling in Syncro as the jobs do not start when scheduled. For example we have a script that places assets in maintenance mode for 4 hours but they start about 4 or so hours late so the script is useless due to the lack of a real job scheduler that starts scripts when intended.

Once Syncro adds a true job scheduler to the RMM solution this will all work but for now it does not.

If there is a way to guarantee jobs start as intended, please let me know.

Richard