As many of you must have noticed, either by visitng or by my desperate tweets, my sites have been down yesterday for almost 12 hours. This is due to what i’d like to call a mistake of mythical proportions. But why am i writing this post now? Well, for starters i want to let you guys know what happened yesterday and, secondly, because my mistake can help you out and warn you, so you won’t have to deal with the same problems i had. As they say: “It’s a very good thing to learn from your mistakes but even better to learn from others’“. So, here is my story.
All my sites, details of which you can find on my sites index here, are hosted on what we call a Virtual Private Server (VPS) – more on VPS’s here. Yesterday was cleaning day. I opened a console to the machine and started tiding up things here and there, cleaning logs, unneeded backups, folders etc. When i was done i wanted to update my machine to the latest software available. The OS that hosts my sites is a Debian Linux distribution so, all needed, was a simple command “apt-get upgrade“. What this does actually is very simple. It checks online to see what versions are available, crosschecks them with the ones already installed and decides which need update. This way everything is done automatically. When i did that, it told me that some packages shouldn’t be updated. That puzzled me a little and i thought “well if i am to update i shall update” so i forced a total update. Process went smooth. All was done. Literally. When the update ended i wanted to check out that all was fine, and guess what, it wasn’t! My web server was unresponsive, it wouldn’t bring up my virtual hosts (thus my sites where all down) and generally it had a mind of it’s own. I started trying to revive Apache but he was still refusing any help. On the process i discovered that it all started by a serious misconfiguration the update caused to the very basic files of the virtual hosting. The damage was irreversible. All was left was a clean install. Fortunately my hosting company offers a very easy re-install process of the OS. The big thing though was backing up, restoring and reconfiguring the server. I had the opportunity to take all the backup i wanted but since time was pressuring i didn’t have a clear mind to remember everything i wanted plus some of the things i did want were too big to backup at the moment so i decided to sacrifice them. All in all it was a very long process that took almost 7 hours to partly complete. The sites are up, just a few administration tasks are there left to complete.
Now, what is this all worth mentioning? Well one big thing for me. Create a backup plan well before you need it. Because when you do, even if you are given an opportunity to take it (like my case), you are going to have loses. That is because on the time of a crisis you don’t have a clear mind to decide and remember what you need to keep and what doesn’t matter. Backup plans are a product of mature thinking and everyday tasks rather than panic reactions. So, my advice, think that your site or server is going down now, what do you want to save? Thanks for sticking around all!