Network Failure

September 12, 2010 Christopher
Greetings all, I'd just like to address the intermittent downtime we had Sept 12, 2010 from 0:00 to 16:30 PST.

We utilize datacenter services from Steadfast Networks ( and they are absolutely great for the most part, however during the scheduled replacement of their core routing equipment some unforseen problems occurred that lead to downtime, slowdowns, and a host of other problems. They are still working on fully repairing the network and getting the new hardware in place. We made some changes in order to mitigate as much downtime as we could and keep the site up and running as much as possible and for the most critical parts of the site. We are still running on this configuration until we get the all clear from Steadfast at which time we'll be back at 100% capacity.

I'd like to apologize to all of our users for any issues they experienced, as you know we normally have very fast response times and Steadfast is a big part of this and we will be working with them to mitigate the chances of this ever happening again.

We were keeping users up to date with any info that we could get our hands on through our Twitter account, we don't use it as much as we should but we are going to try to change that

Update: There is still intermittent packet loss in the datacenter, and the source of the problem is being replaced.