January 12th, 2002

Status Update

It's been awhile since my last post here so I thought I'd let everybody know what's up...

For all of you, it probably seems that not much is up, actually. There hasn't been anything new in awhile, the site keeps getting slower, and all the new features we keep promising "soon" never show up.

Tell me about it. *sigh*

But here's the basic problem.... everytime I'm just about done with a project, be it S2, the directory rewrite, a good documentation system etc., the site breaks.

So earlier this month I thought: Screw this, I'm going to make redesigning the site's architecture into something that can scale beautifully a personal challenge and I'll work on nothing else but that until it's done. After that, I could get back to all the other projects.

Because really, the site working and working damn fast takes priority over everything else. Who cares if we have a super easy & powerful style system or a directory if you can't load your friends page?

So, I've been working on the new architecture the last couple weeks. Mostly, the challenge has been converting the old database format into the new one. After some test runs on a dev server and some math, I found it'd take several days to convert all of LJ. Several days of downtime would suck, even if we had read-only access enabled during that time.

The other day I came up with my 3rd plan of attack for conversion. The necessary downtime now is only a minute and 30 seconds, about.

That code is all written now and tested on a copy of the real data.

The current project is converting all the affected code on the site to understand the new db schema, which isn't too tough. A pre-req for that project is a tool to convert somebody from the old schema to the new schema on some other cluster. I wrote that tonight and have tested moving my own journal (well, the copy on the test server). That's all working. So now that I have data in both the old & new format, I can proceed with making the code on the site understand both types.

Then, for a week or more, we'll slowly be converting users one at a time to the new format. While your journal is being converted you won't be able to post new things and people won't be able to comment in your journal (you will, however, be able to comment in other people's). The downtime per-user won't be more than a few minutes and you'll probably not even notice it. If you do, you'll get a message that says your journal is in read-only mode.

Anyway, this is getting longer than I'd intended.

In summary:
  • our architecture sucks now
  • it won't suck shortly
  • after that, fun stuff.
Sorry for the less than perfect service as of late. I hope you understand that I'm working as hard as I can to make this site permanently rock.