by Andrey » Sun Feb 26, 2006 12:26 am
All,
First.. I'd like to apologize that this maintenance took way longer than expected.. There was no excuse for it, even considering the circumstances..
Second.. I'd like to thank everyone for their patience and understanding.
Third.. I'd like to say that TSN will try to it make up to you guys, but I'd leave it up to Bernie to get more details to you.
Fourth.. I'd like to thank Bernie for his hard work, and [b:0a53c09f9e]his[/b:0a53c09f9e] patience.. He was working days and nights with us to try to find workarounds and alternative solutions.. He's going to continue working through the night trying to get teams up to speed and clean up.
Fifth, I'd like to thank everyone of the crew that participated and helped with the maintenance, and didn't give up after being up for a number hours that I am sure many folks on these boards never experienced.
Sixth.. I'd like to thank everyone for their support and concern about my hand/finger.. After a couple of stitches and a few hours in the hospital I was able to help the maintenance to come to a completion. I am sure a visit to an orthopedic will get me to full strength asap.
Lastly, I'd like to shed a bit more light on the actual maintenance.. This was no simple server upgrade, but a full scale datacenter move involving dozens of peieces of equipment. We were planning it for a number of months, unfortunately it wasn't 100% executed according to plan, unfortunately we also had little in performing the move at a different time, since the datacenter is shutting down this weekend. The last datacenter move I participated in involved more then two dozen people, and a complete mirror of the environment. This move involved far less resources (but was at a similar scale), although we were able to mirror (and upgrade) a lot of the environment (but not all of it) at the new datacenter prior to the move. Since a lot of things were physically moved (some servers are a few hundred pounds) there were plenty of places for the move to go wrong.. You must also understand that there is also little room in 'rollback plan' when your servers are out of racks and in a truck..
Anyway, I won't try to come up with any more details that might seem as excuses.. I feel we dedicated all the limited resources we could.
I'd also like to add that those saying that banks never go down, should visit their bank sites as often as I do (citibank's universal card site was down just a few hours ago when I tried to pay my bill).. Those stating that ebay never goes down, know better.. I believe a couple of those stories can be found in news archives..
Once again.. I'd like to thank everyone for support and understanding. We try our best to keep you happy, we really do..
-Andrey
P.S.
Those feeling that strat games are the last to get a look are not correct. There were plenty of similar problems with servers other than the ones running strat backend, they were just less visible to customers.