When things get extremely wrong

As many of you must have noticed, either by visitng or by my desperate tweets, my sites have been down yesterday for almost 12 hours. This is due to what i’d like to call a mistake of mythical proportions. But why am i writing this post now? Well, for starters i want to let you guys know what happened yesterday and, secondly, because my mistake can help you out and warn you, so you won’t have to deal with the same problems i had. As they say: “It’s a very good thing to learn from your mistakes but even better to learn from others’“. So, here is my story.

All my sites, details of which you can find on my sites index here, are hosted on what we call a Virtual Private Server (VPS) – more on VPS’s here. Yesterday was cleaning day. I opened a console to the machine and started tiding up things here and there, cleaning logs, unneeded backups, folders etc. When i was done i wanted to update my machine to the latest software available. The OS that hosts my sites is a Debian Linux distribution so, all needed, was a simple command “apt-get upgrade“. What this does actually is very simple. It checks online to see what versions are available, crosschecks them with the ones already installed and decides which need update. This way everything is done automatically. When i did that, it told me that some packages shouldn’t be updated. That puzzled me a little and i thought “well if i am to update i shall update” so i forced a total update. Process went smooth. All was done. Literally. When the update ended i wanted to check out that all was fine, and guess what, it wasn’t! My web server was unresponsive, it wouldn’t bring up my virtual hosts (thus my sites where all down) and generally it had a mind of it’s own. I started trying to revive Apache but he was still refusing any help. On the process i discovered that it all started by a serious misconfiguration the update caused to the very basic files of the virtual hosting. The damage was irreversible. All was left was a clean install. Fortunately my hosting company offers a very easy re-install process of the OS. The big thing though was backing up, restoring and reconfiguring the server. I had the opportunity to take all the backup i wanted but since time was pressuring i didn’t have a clear mind to remember everything i wanted plus some of the things i did want were too big to backup at the moment so i decided to sacrifice them. All in all it was a very long process that took almost 7 hours to partly complete. The sites are up, just a few administration tasks are there left to complete.

Now, what is this all worth mentioning? Well one big thing for me. Create a backup plan well before you need it. Because when you do, even if you are given an opportunity to take it (like my case), you are going to have loses. That is because on the time of a crisis you don’t have a clear mind to decide and remember what you need to keep and what doesn’t matter. Backup plans are a product of mature thinking and everyday tasks rather than panic reactions. So, my advice, think that your site or server is going down now, what do you want to save? Thanks for sticking around all!



30 Responses to “ “When things get extremely wrong”

  1. PC says:

    Recently something similar happened to me. My own Host suspended me.

    I know it sounds silly, but it happened and I don’t even have a back up. I have to do a lot of things overnight in order to make my site up, the next day.

    I’ve been constantly backing up my database now and hope everything goes well for you and for me.. :roll:

  2. Raju says:

    Its good that everything is fine now, I will right away ask my VPS host to see if there is some automated backup process available, if not I must plan for something. Probably one lesson to be learnt here is – if we have an automated process setup to do a job, we must leave the job to that. What say?

  3. Rockstar Sid says:

    That is the reason I stay away from VPS :???: coz I dont understand a bit and things get worse when I try to understand :roll:

    But I seriously need to shift one day or the other and fear what if this type of situation may occur :sad: Glad you resolved them!

    Nice to see you back ;)

  4. Shirley says:

    Wow, this sounds like quite a bit of a nightmare! I am glad that you are back up. And it sounds like you’ve learned some valuable lessons. ;-)

  5. stratosg says:

    @All Thanks for coming back and supporting ;)
    @Sid If you need any help i’ll be glad to!

  6. Good to see that you resolved the problem. One of tha main reasons i am not shifting to VPS is because of such issues that can happen. But when i do, i will sure ask for your help

  7. Sire says:

    I’m glad you have it all fixed up Stratos. I am in the process of doing automatic updates of all my blogs, just in case something happens.

    One of my hosts has moved me to another server because they say that my usage is not suitable for shared servers. The problem is that I don’t know why my usage is so high. I have updated all my plugins and the wordpress is the latest version. I’ve given them admin access so they can check it out and they say all seems fine and I have to wait 24 hours for the next cumulative stats. At least they didn’t suspend me.

  8. Sire says:

    PS, I forgot to thank you for your last comment. :roll:

  9. stratosg says:

    @Sire No need to thank for commenting. As i said before i comment when i feel like it and with your blog among a few others i feel like it a lot ;)

    As for your sites… What part is overloaded? CPU? Memory? Do you have any kind of caching enabled? If you want any help you know where to find me… I’m always there for a friend ;)

  10. Sire says:

    At their request I have installed Super-cache and have it running on all my blogs. They say I am using too much CPU and Top Processes, but I have no idea what can be causing it.

  11. stratosg says:

    @Sire: Well since the configuration of the webserver is theirs my bet would be on DB queries. If you want we can take a full mirror of your blog install it on a machine and start hitting it. Disabling plugins one at a time see what is using up all the CPU. It must be the plugins…

  12. Sire says:

    I don’t want to put you to any trouble stratos. Are there any plugins that have a known issue?

  13. stratosg says:

    @Sire: Let me put it this way. If you feel comfortable and trust me with your data in my hands then i’d be delighted to help you. This is the only reason i was hesitating. Else give me a list of the plugins you are using (mail me if you feel better) and i’ll check them out see what happens…

  14. Sire says:

    Statos, I have no worries at all about you looking at it. Email me what you need and I would appreciate any help you can give.

  15. Sounds horrible :( A backup plan is SO important. Good thing you’ve found gog.com to distract you from your troubles ;-)

  16. stratosg says:

    @Sire I’m mailing you know…
    @Kim Thanks to you i did! I appreciate you stopping by Kim!

  17. Sire says:

    Thanks for taking the time to help me out Stratos. :smile:

  18. [...] if it has any effect. Luckily for me Stratos offered to help after I left a comment on his ‘When Things Get Extremely Wrong‘ post. Knowing how smart Stratos is, as he’s developed Blogger Buddy as well as some [...]

  19. stratosg says:

    @Sire: As i said on your blog if i come up with anything more i’ll let you know. Also, if you need anything else you know my mail ;)

  20. Sire says:

    Yes, I do stratos, and I assure you I won’t be a pain in the proverbial :grin:

  21. stratosg says:

    @Sire: I’ll be glad to help out… Don’t feel that way ;)

  22. Raju says:

    @Sid, Madhur and all those who are saying VPS is troublesome,
    you are completely mistakes guys. There are 2 kinds of VPS – managed and unmanaged. What stratos is using is Unmanaged VPS, but what I am using is Managed. I control everything using cPanel & Power Panel… way too easy!!
    For those who are not comfortable with linux, i suggest to go for VPS. It rocks!

  23. Raju says:

    I meant “managed VPS” in the last line.

  24. stratosg says:

    @Raju: This is a very nice thing you cleared out man. It is true that there is a difference and managed VPS is very easy to administer…

  25. I can imagine your situation and relief and happiness you got after everything is restored to normal. :)

  26. Khaled says:

    Glad its back up and running. Is it just the database, plugins, .htaccess and wp-config.php you need to backup for a blog, or do you need to download the whole thing to back it up? I have the database backup plugin installed, which emails me a copy of the database each day, not sure if you use it but its a pretty useful plugin.

  27. stratosg says:

    @Khaled: Well backing up your database is half the job. In order to be 100% sure you need to backup the whole directory structure from your FTP. This is how i do it…

  28. Funny Stuff says:

    It’s good to see you got it back up though.

    A backup plan is important, I don’t do anything that would make huge differences to mine without first making backups and making sure I can fix anything I may screw up.

  29. stratosg says:

    @Funny Stuff: Well you are absolutely right but there are certain points in history that you do a majorly stupid thing without even noticing…

Leave a Reply