Yesterday, at 11:00 AM PT, we released Rich Text Editing for Tickets. Upon release, despite over a month of testing, we discovered new issues. As a result, we rolled this feature back. We recognize that the update affected your core workflows and the issues were heavily disruptive to your business, for that we are deeply sorry.
Many of you have commented that we had previously committed to not scheduling maintenance windows on weekdays. This feature rollout did not require a maintenance window so that policy did not apply in this case. We will be conducting a review to improve our release policy for feature releases that impact core workflows so that this does not happen again.
We intend to resolve the issues that were discovered, and then re-release Rich Text Editing to a small group of early adopter users before more broadly rolling it out. If you would like to be in that group, please email me at ian@syncromsp.com. We will also issue an update to our general release policy to limit impacts to your business and share it with you in the coming weeks.
As a result of the rollback, ticket comments and leads that were created between 11:00 AM and 3:32 PM PT will display HTML in the UI and in emails that were sent during that window. When we re-release Rich Text Editing, this HTML will render and the issue will be resolved automatically.
If you are still seeing issues in your account, please contact Support so we can assist you.
Force updates either go super right - or super wrong:)
I love the idea of the update - but I think many were a bit blind sided by the addition. While bug fixes and security changes are almost a must - I say at least bigger updates should still follow the normal roll out. This way if there is a bug - it should be way more noticeable before release. Even then - start with customers that are ok with some type of testing (or hot flash updates) then follow by delay release of production customers that are ok with getting them early then finally everyone else.
Thank you for the clear and concise response exactly what was needed given the number of community post this roll out generated.
I will take your word that there was a month of QA around this update but at the end of the day it was still missed.
In addition to what Travis has said I would like to put my own idea forward.
Follow the Microsoft approach yes still do internal testing (don’t just fire the QA team like they did) but implement an insider program for any and all updates.
Have users opt in and test genuine use cases to discover these issue. Yes less convenient from a dev side but ultimately more successful as it’s prevents a massive backslash. - not saying it won’t happen but should reduce the impact of big bugs being shipped.
I think any new feature should be released in small batches of users. This way the effect of something breaking is limited. Irrespective or requiring a mantenance window or not. Fully understand your making improvements and we love to see this. It just needs to be tempered with a softly sofly approach until you have the confidence its good to be rolled out en mass.
Syncro really needs to wake up regarding the impact this has on its users. We are incredibly sensitive to this considering the chaos from the attempted platform change in April of this year.
Please make these decisions based on how would you feel if this impacted your MSP business.
I don’t know if this is doable, but when CW did the HTML feature, we went into our settings and enabled the pilot. We could disable and enable at will for a couple of months until it was fully released. This allowed us to test and if things were really bad, report, and undo. Now they still released it broken, but that’s beside the point :).
These things happen, What I think this shows, is that the company made a good choice in how this was handled. It was reverted back with quick ease… I don’t think many users here understand how difficult that may of been and to even HAVE that version history. Sure it’s more common in this world to have version control, but all these different resources as back up have given me a confidence in the Syncro team.
Some companies can’t even admit when they break something…
Kudos to the fast rollback! Thank the programming gods for GIT and version control (if you all even use GIT, but same difference)
Most people are just upset that it didnt go through some normal means. Half way through the day it just show up without warning and then they notice issues across the board.
While I agree they did make fast work to rolling back, any UI / fault issues that can break workflow at this scale can be problematic for a business. We could be talking serval thousands of dollars of lost and or reflect an issue on their end to look like an issue with the MSP provider.
As I said before in another thread - I been around to all the MSP platforms - so I know none of them are perfect, but more so, common practice says to release in small groups before releasing to everyone. This way if there is an issue it should’ve been caught sooner. I think this is what people are most upset about than the issue it self.
If it was me - I would offer a setting in the admin page to opt for different deployment updates. For example, customers that NEED always up and operating time frame would set their upgrade path to “Slow, and Study”. This would mean they always get the feature update (anything that changes UI or design) last or at least a few weeks out. On the other hand, customers that want to be some place in the middle (default) would be in the next release or at least a week out. Customers that don’t mind and always want to see the newest features/updates could be in the Fast Path. Customers in the Fast Path would need to agree to a few things - but would need to offer a way to roll back if needed. Fast Path maybe get some other perks for being first and dealing with the issues first ~ but that just my 2 cents.
Or even a test environment we as subscribers have access to. Similar to in gaming when you have a PTS (play test server) that is separate from your production environment, and basically demo the features and report what’s breaking. This SaaS company has a great community and could very well set up an environment to those who love trying to break things or just play with new features and also utilize this forum platform for opinion on the changes so that they can also get a community opinion before an official roll-out.
My 2 cents on the HTML in general. I commented on this a while ago in a thread about this. I was concerned about the real-estate that this feature would take up on the ticket page. Happen to turn out as I expected unfortunately. While you are back at testing this feature, please make the box expandable as the way it was released takes up way too much space on the ticket page.
Hi Gary, I was asleep on the other side of the world when all the fun happened.
I take it they have changed this? What extra real estate is it taking up. The ticket window is pretty busy, really would not want it more difficult to navigate for the sake of HTML
I don’t have a screen shot of it. But suffice it to say that the HTML window takes at least twice that space. Also I had a customer with all kinds of crazy “fancy” formatting in their outlook and a two sentence reply I had to scroll to see the full comment section of that one reply.
IMO - ticketing is down and dirty, knock em out as efficiently as possible.
I was thinking about this - but if you break down what happen… I dont think some of this would’ve been seen unless they make an active test environment with all functions being used by some type of bot. For example, having a bot send in tickets, chat request, leads, etc. Not saying its impossible, but would have to be an active PTS vs a static one. If you are going that far - then they wouldn’t really need us to break things - another bot could in theory still perform test/actions to make sure everything is correct.
I, for one, absolutely love the concept and have been asking for this since signing up! I would, however, expected the feature not to be enabled by default and instead have it as a setting in the admin section with potential permissions for staff who do/don’t want/need it type deal.
While their current version of crowdsourced trial by fire seems to be working for them, so I don’t expect them to change it any time soon (not to mention they seem to be just getting better at performing on-the-fly rollbacks) but a group of testers would have caught at least the major issues within a few hours or so of testing, right?
If its taking twice the space (length wise) then it would have displaced the customer info section to the left of the communications section. So we have to scroll more ?
The layout changes depending on your screen space… so while in some cases it could, but in others it wouldn’t. So a mock up wouldnt really work for say only because its going to depend on how big or small your relative space will be to display everything. The default spacing was larger though than it is now and that section just got a bit busy over all.
