Website Down!

Website Down!

Postby Varyar1G » Sat Mar 23, 2013 11:37 pm

So...
Many of you may have noticed that the website went down yesterday for quite some time. We apologize for the technical difficulties that kept you from this, your home away from home. However, we believe we have rectified the issues that plagued us earlier and hopefully we won't see anything like them again in the future. If we do, we should be able to fix it much more rapidly now that we figured out how. I want to take this opportunity to publicly thank teh_leet_haxor and biomedalchemist for their tireless work today. They quite literally went above and beyond what any of us could have expected in getting this place back up and running. Also, I would like to say thank you to those in the irc who watched my slow descent into madness as the site remained down and sort of attempted to keep me calm. Except for Shawncaster. He was egging me on in my threats to burn down datacenters while Sekani asked me to post YouTube video if I ended up losing it and kicking some customer's kid through a plate glass window (I was at work when all this went down). Fuck you guys.

Also, to Neal from GoDaddy, I hate you and your attempts to reach into my wallet before you would fix something I already paid for. I wouldn't piss on you if you were on fire.

To Kyle from GoDaddy, we appreciate your patience while you tried to help us fix things. You failed miserably and didn't know what you were talking about, but thanks for being willing to talk to us for an hour and a half at least trying to solve the problem. Good Game and other cliches to that effect.

To the rest of you, this is why we're running the open beta, so I guess you get what you pay for!

That is all
Varyar1G
"You cannot exaggerate about the Marines. They are convinced, to the point of arrogance, that they are the most ferocious fighters on earth - and the amusing thing about it is that they are." - Father Kevin Keaney - 1st MarDiv Chaplain - Korean War
"Despite what your momma told you, violence does solve problems." - Ryan Job
User avatar
Varyar1G
Founder/Owner
 
Posts: 487
Joined: Sat Mar 02, 2013 8:26 pm

Re: Website Down!

Postby teh_leet_haxor » Sun Mar 24, 2013 7:44 am

As part of Biomed's initiative to just say what actually happened, here's the outline of the problem, remedy and future plan:

While I was out, something must have fiddled with the server at the fundamental level, in a way that stopped its DNS service and broke its configuration file such that it could not restart. This part remains a mystery for now, but I can't think of many possible perpetrators beyond GoDaddy themselves. In any case, this was where a GoDaddy support representative took a look, then asked Varyar for $50 to fix it, which is the most bizarre behaviour I've ever heard from a web-host. Varyar wisely declined.

Upon my return, Varyar was amid cursing GoDaddy for their foul knavery and eventually got me to talk to them instead. I first went through actually getting us root access, which is something I've been seeking for a while anyway. After that, with me in a rooted SSH terminal and Biomed ready to respond with ideas and expand my half-knowledge of linux commands, we went through various checks and eventually took a stab at replacing the DNS service configuration file with a fresh/default one. This allowed the DNS service to start up again, though without serving any records. I did try editing and re-saving the DNS setup in the web control panel, but what eventually got it working was Varyar pressing 'Restore defaults' and re-adding the DNS records from a screenshot of how we had it.

We do need to change this setup somewhat, mostly because it's still a fairly bad single point of failure at the moment. Domains usually specify at least two nameservers to guard from the failure of one, so Biomed and I are going to look through some ways to set up some external name serving.

As Varyar says, this is why we have a beta month.
User avatar
teh_leet_haxor
Site Admin
 
Posts: 615
Joined: Sat Mar 02, 2013 7:39 am

Re: Website Down!

Postby BiomedAlchemist » Sun Mar 24, 2013 12:48 pm

In Shawncaster and Sekani's defense, we all thought the video idea was awesome. :P

This is the actual error we saw on the server:

# service named status
WARNING: key file (/etc/rndc.key) exists, but using default configuration file (/etc/rndc.conf)
rndc: connect failed: 127.0.0.1#953: connection refused
named is stopped


The customer service rep who worked with teh_leet_haxor kept telling us about a fix-it script (/scripts/fixrndc) and other ideas, but all of them only applied to customers who use cPanel (GoDaddy's normal control panel service). We upgraded to Parallels Plesk, so most of his ideas did not even exist on the server in any filepath. It would be kind of nice to know that the customer service rep can read us the right script after pulling up our account and seeing we do not use cPanel. ;) Anyway, I prefer to Neal's initial response (summarized, not directly quoted): "You must have screwed it up despite having the necessary root access to alter any of those files. Give me $50 and I may be able to fix it, but it could cost more if it takes me over 30 minutes."

In the future, we should be able to set up a free secondary DNS so that if the primary goes out (as long as bad data isn't propagated), the site will still be available for all of you to enjoy.

-BMA
User avatar
BiomedAlchemist
Veteran
 
Posts: 371
Joined: Sat Mar 02, 2013 8:48 pm
Location: Long Island, NY


Return to News

Who is online

Users browsing this forum: No registered users and 2 guests

cron