I’m about to walk out the door to catch a movie this evening and I get an SMS alert from Pingdom that my server is down. “That’s strange, I haven’t even logged into the server today,” I think to myself. I hop on my Mac to check things out and it definitely is down. I hurriedly attempt an emergency console connection but that doesn’t work. Not being able to do anything nor be late for the movie, I shoot off an email to the Slicehost support staff that I can’t login to my slice and head out the door. Half way to the theater I get a response that the server experienced a drive failure but for some reason did not gracefully failover, requiring a hard reboot, and that all slices were running again.
I get to the theater, settle in, and immediately whip out my iPhone to make sure everything is operational. Static pages are loading, but PHP pages such as this blog result in a gateway error. I emmediately know the problem: php-fastcgi did not get started when the server was rebooted. I was half tempted to send Slicehost an email asking them to run the start sequence, but, I did not want to compromise my root password and deal with changing it again. Instead, I decided to relinquish the $5 required to buy an SSH application for my iPhone. To my surprise, I found a free app, TouchTerm. The movie previews were almost done so I clicked install, waited a couple of minutes and finally launched the application. To my surprise the application worked pretty well under the circumstances. I was able to send my server the critical command to get PHP back up and running, just in time for the opening scene.
I’ll definitely be keeping this little app handy for the next emergency.
I do find it amusing that a legitimate, unscheduled down time occurs just days after writing about uptime monitoring with Pingdom. It was nice to know that everything worked like it was supposed to and that I was immediately aware of the situation, even though it was out of my control.