That time when Digital Ocean destroyed my site
It shouldn’t come as any surprise that this isn’t my only blog. I mean, with the high quality content you get around here, of course I have years of experience!
It also isn’t really a secret that I host at DigitalOcean. And yeah, I used a referral link, even with a headline like that. Even with a story like you’re about to hear. Because I’ve been doing this for a long time and the tl;dr version of the story is, I still think DO is the right host. They’re cheap, they’re fast, and for the most part, they’ve been reliable. Although DO’s not totally without fault here.
Anyway, back to the story. So I was doing an upgrade of my other site, and I needed to play around in the database. I’m not a DBA, so I wanted to make sure I had a backup. In fact, this was a live and heavily trafficked site, so I wanted to spin up a test server to make the changes on first. DigitalOcean makes this super easy and super quick. All I needed to do was go to my backups and click “create droplet from backup”.
Ah shit. That is NOT a good sign. So I did what anyone who is *PAYING * for something they’re not *GETTING *would do: I opened a support ticket.
A very nice man came back on the ticket and told me there was some kernel mismatch and I needed to do a hard power-off of the machine and boot it up cleanly. Alright.
So other than that, I hit up my other site and was greeted by… my other site. With one noticeable exception… it had reverted to FOUR* ***ING MONTHS AGO. Which coincidentally, was the last time the backup had run successfully. It’s a 1GB plan, you can’t tell me four months worth of semi-daily posts had failed to write to disk?!
So I updated the support ticket. A nice friendly and very helpful response came back. “Oh geez that sucks. If there’s anything else we can help you with…”
Seriously, it wasn’t much more than that. A quick line of “we don’t know why that would have happened” and then nothing. So I did what any technical person would do. Backups are running now? Let’s mark the ticket as closed.
I never did figure out what happened, but luckily since it was an upgrade and I’m not a blithering idiot, I had a local copy downloaded to my machine. But there’s one more thing… my other site runs on WordPress, and I used Updraft to keep a second set of backups (again because I’m not a complete moron). That’s my savior, right? Wrong. Even though it was set to keep two backups and run daily, here’s what greeted me:
That’s right, a backup done after the site upchucked, one done in June (coincidentally, the same time my DO backups failed…) and one done in May when I first launched that particular site.
So what’s the moral of the story? I guess keep three backups in case the first two fail? But seriously, even though this really sucked and couldn’t have come at a worse time (and why the **** did my backups fail and no one let me know, DigitalOcean?!), the infrastructure around DigitalOcean made getting everything running again a breeze. Spinning up a dozen new servers to try things out took just a few minutes, the one-click WordPress app was a breeze, and their servers are FAST.
Plus, I was using Linode before I switched to DO and at least DigitalOcean has never lost my *****ING CREDIT CARD. *And then *LIED TO ME ABOUT IT. *Two GOD DAMNED TIMES. LINODE. YOU ASSHOLES.
So yeah, use DigitalOcean. They’re Not That Bad™