Jump to content

A Day Without HAIF


editor

Recommended Posts

Let me start by apologizing to everyone out there for the HAIF downtime. It's sad when tens of thousands of people are turned away from a web site by an error message. It's worse when I can't even replace that error message with something more meaningful. Thanks to everyone who e-mailed, Tweeted, and Facebooked their concerns. Here's what happened:

We upgraded the server that runs HAIF. It also runs my other 15 web sites, most of which are very low-stress compared with this forum.

During the upgrade process, all of the databases lost their passwords. I had to go in and reset each of them to what they should be. As I did each one, the web sites that depend on those databases came back online. Except HAIF.

For some reason, the database that runs HAIF wouldn't accept the new password. Or at least that's what it looked like. In reality, what was happening is that the database was pointing to an entirely different server. The server where we used to be back when we last upgraded in 2006.

So when the 2011 version of the HAIF software tried to wake up, and saw a tremendously outdated database from 2006, it fudged its Huggies, leaving all of you with the error message we saw earlier today.

It took a while to track it all down, and eventually the solution was to just drop in the backup of the database I made last night. So, HAIF as you see it today has about a 16-hour memory gap. But I guess that's better than starting over with the messages from September 13, 2006.

As a point of interest, the new server is about 8x more powerful than the old one. Same memory as before (the entirety of HAIF still fits in 500 megs of RAM), but we're now running it on a processor from 2010, instead of one from 2002, and on a modern motherboard with all of the cache and bus goodness that that entails. Oh, and we're now RAID 1, in addition to my nightly backups.

For those of you who are Unix/Linux fans, our load used to be around 1.75-2.10 on an average afternoon. Right now it's running from .02-.15.

Hopefully everyone notices and enjoys the speeds. Right now, HAIF pages are rendering in the server in a little under a tenth of a second each. It's getting them out the door that's the bottleneck now.

Link to comment
Share on other sites

Archived

This topic is now archived and is closed to further replies.

×
×
  • Create New...