Update on System Maintenance from April 12 2022

*slow clap"… lost so much!!

Fortunately, we use our own SMTP server, and we were able to view our sent items folder for updates between those times. Hope that helps others.

2 Likes

I actually had no idea we had service issues, as my portal was loading fine all night/this morning. I was actually actively answering tickets and submitting information this morning.

Thankfully I have ticket reports via e-mail, but other items are missing of course. And the confidence level/expectations are hit. It’s not the restore, but the complete lack of quality control and communication.

3 Likes

oof, sorry for your loss man.

all I’ve seen since the start of my day was 504 pages and the dashboard, but we’ve had a pretty slow morning and can go through our email logs on our SMTP server to see whats happened. Guess we got off easy today.

1 Like

Hi Miles!

This should be resolved now as well. Please re-test and let us know!

Yep, we have data loss on this.
The system came back online a few hours ago, and I imported a bunch of notes and time on items worked on while the system was offline. It all saved fine. Now it’s gone.

The business hours downtime was annoying (APAC user here), but the data loss without warning is downright frustrating. Believing that our data was safer with a larger provider was one of the reasons why we moved from on-prem RMM to cloud provider RMM.

4 Likes

:+1: all good now, thanks

1 Like

Are you referring to the system being stable and is the data loss currently being addressed?

?? Dude, is the thumb like an up yours, cuz that’s what this deserves.

1 Like

Yes - the system should be stable and running as expected, with the exception of some data loss if it was entered during this extended and unexpected outage. We are continuing to monitor all incoming tickets and system statistics to ensure complete system performance.

This is unacceptable. We have lost tickets, charges, estimates for the whole of our working day. We could at least sporadically view the site before you rolled it back, if we knew you were going to roll it back we would have noted down the day’s input so we could re enter it. Not impressed at all.

2 Likes

Your time frames are wrong - the restore was from 6a eastern yesterday.
You burnt 27 hours on us without warning.

1 Like

For us the API is still not working (in UK). Also we have lots ALL tickets and work in Syncro from today. Not very impressive!

I still have comments from 4/12 at 5pm EDT.

1 Like

I have some leads but lost all of my tickets for the day of the 12th. It sends me emails when I create, so I’m not dead yet.

Does anyone else see scripts stuck as pending?

Due to extended instability after our maintenance window yesterday we made the difficult decision to restore services to the state prior to the maintenance window. We explored all avenues for complete service restoration before taking this action.

We know that you trust us to run your business, and we will be performing a post-mortem and sharing information later this week on tonight’s events.

Any data created during the period from 11pm PT April 12th - 6:30am PT April 13th was reverted.

If you have critical data that you created during this time, please contact Support for further assistance.

A note about payments: If your credit card payments are processed by Syncro, please check your processors in order to manually update your Syncro records accordingly.

If you have any specific questions, please contact Support and they can assist you.

Hi Pete.

Our restore was from right before we started our planned maintenance window, not 6am on 4.12.

If you’re seeing data loss that is from before the maintenance window please open a support ticket so we can investigate further.

Thank you!

Does that mean there might be a method of restoring lost data?

Very good question @BrianMorris :+1:

@anon79052621 … Care to comment? This would really help a ton of people and prevent the need for tons of manual input.

1 Like