We are investigating intermittent issues with our platform. Some users are reporting error messages while others are experiencing slow access times.
Updates are being published to our status site at https://www.syncrostatus.com/
Please follow the status site for the latest updates. For fast access to the status, you can check the status here in the community using the status widget.
@anon24463327 In case it wasn’t on your radar yet, queueing to our own smtp is also slow. Sending invoice emails or replies to tickets notifying our techs or the clients are all delayed in getting over to our smtp Mailgun, as soon as they arrive at mailgun they are sending immediately, so the delay appears to be on the Syncro side. Let me know if you would like more detailed information that I have gathered.
Our development team has implemented fixes for the sluggishness issues experienced this morning. We continue to monitor the situation and are seeing improvement, although some accounts may still be affected.
The team is actively working on the issue until it is fully resolved and updated will be posted here and on our status site.
@joel1 I appreciate your update. Most services that have to be queue are being affected. It should be improving soon if it hasn’t yet.
We are continuing to monitor site performance. We have seen improvements in page load times but some users will still experience delays in various areas of the application. Google Calendar and Quickbooks still have tasks queued and may see a delay.
We’re also still observing 504 errors, is it coming and going in your system? How often are you seeing this one?
It’s been decent with normal ticket work over the last hour. I get the 504 when try to load a Ticket Time Tracking report:
hxxps://DOMAIN.syncromsp.com/reports/ticket_time_tracking?utf8=%E2%9C%93&filters%5Bperiod%5D=Custom&filters%5Bfrom_date%5D=06-01-2022&filters%5Bfrom_date_alt%5D=2022%2F06%2F01&filters%5Bto_date%5D=06-21-2022&filters%5Bto_date_alt%5D=2022%2F06%2F21&filters%5Bemployee%5D%5B%5D=&commit=Run+Report&saved_report%5Bname%5D=&saved_report%5Breport_name%5D=ticket_time_tracking&saved_report%5Bname%5D=
Okay, that’s consistent with what we’ve seen. Ticket Time is affected. We see it spike and it comes back down- development is still triaging the remainder of these.
Also getting 504s on the customer portal and on invoicing when using the ‘add to invoice’ button on pending ticket charges. All time-based everything is still not working.
The frequency should be reduced but you’ll still see it at the moment. They are working on some deployments now.
Our Engineering team has applied multiple updates to resolve slow page times and some error notices throughout the platform. We are seeing performance return to normal and are continuing to monitor the situation. We anticipate full resolution for all customers as traffic continues to normalize. Until that time some customers may still continue to experience service interruptions. We appreciate your continued patience and understanding. Updates will continue to be posted throughout the remainder of this event.
What caused the issues?
Site performance has returned to normal and services have been restored. The system is now fully operational and degradation has been resolved. You may see an interruption here or there but we expect this to clear out as we see a continued resolution of services.
Thanks for your patience everyone. We can consider this one resolved.
@mgiordano I don’t have that information at the moment but if we receive additional details we’ll post them here.
Thanks. My concern is, was this related to the upgrades to the system that were talked about in the other thread?
It would be transparent to provide more information about this slowness and timeout errors.
Was it cased by an upgrade?
What is the likelyhood that the slowness and timeout errors will happen again?
What caused this issue? Please be specific.
Hello again everyone, since I don’t have information on the root cause I cannot speak to the details of what caused it as I am not privy to them. If there is a retro that won’t come from the Support team. If it’s something you’re interested in I do welcome replies on it so you can voice your concerns, I just won’t be able to answer those directly myself.
