Our team is investigating Reports of Slow Access and In-App Timeout Errors

We are continuing to monitor site performance. We have seen improvements in page load times but some users will still experience delays in various areas of the application. Google Calendar and Quickbooks still have tasks queued and may see a delay.

Yep, still issues. This is right now.

We’re also still observing 504 errors, is it coming and going in your system? How often are you seeing this one?

It’s been decent with normal ticket work over the last hour. I get the 504 when try to load a Ticket Time Tracking report:

hxxps://DOMAIN.syncromsp.com/reports/ticket_time_tracking?utf8=%E2%9C%93&filters%5Bperiod%5D=Custom&filters%5Bfrom_date%5D=06-01-2022&filters%5Bfrom_date_alt%5D=2022%2F06%2F01&filters%5Bto_date%5D=06-21-2022&filters%5Bto_date_alt%5D=2022%2F06%2F21&filters%5Bemployee%5D%5B%5D=&commit=Run+Report&saved_report%5Bname%5D=&saved_report%5Breport_name%5D=ticket_time_tracking&saved_report%5Bname%5D=

Okay, that’s consistent with what we’ve seen. Ticket Time is affected. We see it spike and it comes back down- development is still triaging the remainder of these.

Also getting 504s on the customer portal and on invoicing when using the ‘add to invoice’ button on pending ticket charges. All time-based everything is still not working.

The frequency should be reduced but you’ll still see it at the moment. They are working on some deployments now.

Our Engineering team has applied multiple updates to resolve slow page times and some error notices throughout the platform. We are seeing performance return to normal and are continuing to monitor the situation. We anticipate full resolution for all customers as traffic continues to normalize. Until that time some customers may still continue to experience service interruptions. We appreciate your continued patience and understanding. Updates will continue to be posted throughout the remainder of this event.

What caused the issues?

1 Like

Site performance has returned to normal and services have been restored. The system is now fully operational and degradation has been resolved. You may see an interruption here or there but we expect this to clear out as we see a continued resolution of services.

Thanks for your patience everyone. We can consider this one resolved.

@mgiordano I don’t have that information at the moment but if we receive additional details we’ll post them here.

Thanks. My concern is, was this related to the upgrades to the system that were talked about in the other thread?

2 Likes

It would be transparent to provide more information about this slowness and timeout errors.

Was it cased by an upgrade?
What is the likelyhood that the slowness and timeout errors will happen again?

What caused this issue? Please be specific.

Hello again everyone, since I don’t have information on the root cause I cannot speak to the details of what caused it as I am not privy to them. If there is a retro that won’t come from the Support team. If it’s something you’re interested in I do welcome replies on it so you can voice your concerns, I just won’t be able to answer those directly myself.

@Alexandra that is ok if you cannot provide the information on the root cause.

Can anyone from Syncro provide the information, or does no one in Syncro have the information?

Was it cased by an upgrade?
What is the likelyhood that the slowness and timeout errors will happen again?

1 Like

Hi All,

Thank you for patience. We are working internally to provide more details and will update you as soon as possible.

@canden.hicks it has been 7 days.
This isn’t a good look. At this point it looks like Syncro is refusing to provide any information.

1 Like

@canden.hicks

In my mind, this can either mean one of 2 things.

1 Syncro have no idea what caused it eight days after it occurred.

2 Syncro are trying to figure out a way to spin the issue.

This would be highly concerning if it is the former, which I doubt it is.

We have all had issues and did not know the cause of the problem immediately. However, after eight days, not having any information would be appalling in any type of IT environment.

However, I think it is the latter, and the community has repeatedly said that we want more information about these issues when they occur. If the essential information is not being passed along the chain, this is a failure point; this will ultimately continue to be a pain point for the community.

Even saying we believe X Y Z caused the issue in the form of work notes would be something at least, and then have an RCA with all the details later.

Utlamitley people lose faith in a product without full transparency, especially when it affects their livelihood.

2 Likes

Keep in mind with your statement - people can have different issues. While one issues might effect 10 people out of 100 that doesnt mean its hitting the other 80 people the same even though its looks to be the same issue. Trying to say they need more transparency after they said there was an issue and made it publicly available so other know is already transparent to a large degree.

Sometimes being too transparent is also a bad look for your services and products too.

Hello Partners! We wanted to share some details on the incident involving system slowness which took place on 7/5/22. Upon discovering some performance-impacting database transactions which were unrelated to our recent system upgrades, we were able to immediately stop these transactions. Shortly after the system returned to a normal state. While the underlying issue existed for an extended period of time, we have set up monitoring which will allow us to discover new potentially impactful transactions more efficiently in the future. Additionally, we are working to improve our communication process to notify you regarding issues more quickly. We appreciate your candor and patience as we work to resolve these issues.

4 Likes