Our chat rooms were down for more than 24 hours over the weekend, this is the longest down time that I remember. Good things came from this bit of down time, but we were certainly not happen about the unplanned issues! I posted lots of updates on the connection issues page, providing any information that I was able to get, so hopefully many people jumped over to the connection issues page to see what was going on.
Many of our regular users already knew to check in at our cams chat page and pop into the “new chat” system to see if it was something that everyone was experiencing, or a localized issue. I am thrilled that we have kept our backup systems up and running and were able to provide additional options for everyone to keep in touch.
We also provided a link to our upcoming mv-chat system, and some people found that to check it out. We got some good feedback about things we can do better with this new system, but overall it worked mostly as expected. We found a few settings we need to adjust before it is ready for more public testing, and from some of the questions that I fielded in the mv chat system the short times I was there, I can see that I will need to make a little tutorial showing how some things should work with this new system. Information about the login procedures and such will be updated and posted soon. I really appreciate the feedback and suggestions that people provided in the chat room and on the mv chat info page. Look for this to be updated with more info in the near future.
I spent so much time scrambling with our internet backbone connection that I did not get a chance to update our wordpress.com pages, or release information about our new twitter info and other ways to connect with us. I would of liked to have rolled out some info about all that, but those things are already coded into our upcoming update, so it seemed silly to make some temporary additions to this version of the site when we already have all that new code ready to launch with the new version. Had I known that the issues would take as long as it did I would of done a few things differently, but on several occasions we were told “60-90 minutes for resolution”.
Unfortunately most of the suggested time frames turned out to be off by many hours. Our data center peeps are going to be establishing two new backbone connections with a new upstream provider this week, so hopefully we will not have the same kind of issue happen again. I give props to everyone who helped sort this issue out on a holiday weekend, although I have some reservations about a certain company which I will not name that could of provided better communication about the issue and their resources that were available over the weekend to find the problem and get it worked on.
All in all some excellent learning experiences this weekend, and hopefully we will use these lessons to make many things better in the future. Those users who were not aware of our chat connections page or our other chat options, sorry you got locked out for so long. Our web server was running just fine the whole time, it was the uplink provider that connects our data center to the internet that failed.
We will also be adding more options for keeping abreast of what is happening using different portals as well. Looking forward to speeding up some of our new updates and adding some more new goodies to the site soon!