Downtime over last weekend

by **Dan** » Wed 9 Jan 2008 13:20

As alot of you will have noticed, all of Silly Project was down over the last weekend, from Friday evening until Tuesday evening. This was because the main Silly Project server suffered a massive operating system failure during a reboot, which rendered the entire server completely unaccessible. The server refused to boot the operating system, making it impossible to connect via remote desktop or any other conventional means, and obviously this knocked everything offline, the http, ftp, and email servers, all the websites and forums, the LFS servers, the radio server and so on ...

After alot of swearing and head scratching, and giving OVH's useless technical support a right good bollocking for not being helpful, I managed to gain access to the server using a special "Recovery Pro" network boot system provided by OVH, which boots the server onto a special Linux system over the network. Once the server was started with that, I was able to manually mount the hard drives in raid into the Linux system to gain access to the files on the hard drives. Once that was done, I was able to use a special SSH/SFTP program to access the Linux system remotely and recover/download all the files from the server. This took forever, about a day and a half, to get everything back off the server, due to the complicated and slow way of accessing the data on the server, and the fact that there was about 400 Gb of data to get back. I was able to use another server in OVH's datacenter to recover the files with, thanks to my mate Andy who let me have access to it. Once I had recovered the files to that server, I was able to download everything to home so that I had a backup of everything ...

Once all that was done, I was able to completely reinstall the server with a fresh operating system. I also took the opportunity to make some changes in the way the server is setup. The server is now running Microsoft Windows Server 2003 R2 Standard x64 Edition SP2, which is a 64 bit operating system, the server was running the 32 bit version before. This new 64 bit system makes the server slightly faster in certain situations, but more importantly, capable of handling large memory loads. The server is now running the 64 bit version of MySQL 5 database server as well, along with the latest version of PHP 5 as default, instead of MySQL 4 and 5 together, and PHP 4 as default ...

So anyway, after a few nights with next to no sleep, everything was finally reinstalled, configured, and back online on Tuesday evening. There wasn't alot of data loss, only a few corrupted files, but nothing too serious. This is an extremely rare thing to happen, but just incase it ever does happen again, everything is now backed up to my server at home ...

So there you go, that explains what happened ...

by **Stevo** » Wed 9 Jan 2008 13:25

That explains all

... how long did it take for you to write all tht? lol must be the biggest post on the forum ^^

At least it is very rare and shouldn't happen agen

by **Dan** » Wed 9 Jan 2008 13:30

The longest post on the forum? Hmmm, dunno, lol. Took me about 15 mins to think and type everything, lol.

And yes, as I said, it is extremely rare that something this serious happens ...
But looking at the better side of things, I suppose that it was a good learning experience.
At least now I know exactly how to deal with such a problem, and now everything is backed up.

So much for my weekend anyway, fucking server, and it didn't help that I was sick all weekend either!

When something goes wrong, everything goes wrong, it's always the way, sods law isn't it ... :roll: