Wandering Thoughts archives


The sinking feeling of discovering a design mistake in your code

It is not a pleasant feeling to slowly realize that you've made a design mistake down in the depths of your program's core data structures and code flow. I am having that experience now, so this is a war story about how what looks like the obvious right choice can be anything but.

One of the major things that DWiki does is convert DWikiText into HTML. Back when I started coding DWiki, I decided that the fundamental result of this rendering process would be a blob of HTML. After all, what could be more logical? The job of the renderer was to turn DWikiText into HTML, so clearly its output was a blob of HTML.

In retrospect the wheels started coming off this particular wagon almost immediately; I just didn't pay attention to the warning signs. You see, the output of the DWikiText rendering process is not just a blob of HTML. It's also things like the title of the page (in several different formats), whether or not the page is cacheable, what directory modification times are important for caching the page, and whether the page specifies permissions settings. And the giant blob of HTML has important structure itself; for a start, part of it is the title and part of it is the rest of the text, and it would be nice to be able to get one without the other.

(Simplifying, if a page starts with any level of header the header is taken to be the title. But the header is also rendered as part of the HTML and can't be separated from it later. This is why the Atom feed here has always repeated the entry titles in the actual text of the entry; in retrospect that should have been a warning sign.)

All of these extra things that DWikiText rendering produced are sort of glued on the side of things in the current code in various somewhat hacky ways (which should have been a warning sign to me). What the core rendering process should produce is an actual data structure that represents all of these bits explicitly. Code that just wants the HTML blob would then generate it from appropriate bits of the data structure.

Why didn't I see this when I wrote the code? Because I started from the basic operation of 'turn DWikiText into HTML' and never lifted my head up to see the growing big picture. Every time I needed something else from the rendering process I basically got out a hammer and took another shot at the code, because that was the easy way (and towards the end, the thought of making major changes to the rendering process scared me).

Now I've reached the point where that doesn't work any more. To make the long delayed changes to DWiki that I want I'm going to have to totally rip apart and redo core rendering (complete with the instability that that implies, plus I'm going to have to understand the code again; it's been years (and yes, that's a bad sign too)).

(One benefit of the change will be better Atom feeds. Another will be better caching. Right now I'm sort of caching the generated HTML when what I should be caching is the underlying data structure that results from rendering.)

Sidebar: the insoluble problem that pushed me over the edge

When people look at a single entry I want there to be a little discreet entry date below the entry title (I've tried a version with the date above the title and I don't like it). However the entry title and the entry text are all part of the same HTML blob and are currently inseparable; I can't crack them apart to insert template logic for this the way I want to.

There are a pile of ugly hack workarounds, none of which I like. For example I could make something that chopped the title out of the HTML blob with a regular expression, but ew. I could also hack up the rendering process to directly insert HTML for the date (in various ways), but that's equally unclean and also has unpleasant interactions with the disk caches.

All of this would be simple if CSS allowed you to relocate <div>s in the page layout, but as far as I know this is impossible (short of manipulating the DOM with JavaScript). You can fix the position of a <div> in various ways but not say 'slice it out of here and put it right after that thing, then lay everything out normally'. All things considered I can't really blame CSS for that omission.

python/DWikiCoreDesignMistake written at 22:15:54; Add Comment

Thinking about how I use email

I've recently wound up thinking that sysadmins probably use email a little bit differently than other people, so I've decided to write down some words on how I use email (as distinct from the tools I use for this).

A bunch of my email is archives, in a sprawling set of folders. I archive things for a variety of reasons; the major ones are more or less sentimentality (email to and from some people), for active future reference (for example, I keep a copy of the current build instructions for all of our machines), and as a genuine archive (for example, I keep a copy of all conversations with vendors). The important thing about archives is that I almost never look at them; they are not active email. As a result they're not really interesting in terms of email handling.

My active email is in my inbox and spinoffs from it. My habit is to almost always keep entire conversations, start to finish, until the whole issue has been concluded (at which point I'll delete the entire thread). Conversations that I'm involved in or might be almost always stay in my inbox; conversations that I'm just monitoring I try to shove off to a side folder (although I don't usually manage to do it right away; often I hope that the conversation is only going to be a short one, so I can delete it soon). In short and in theory, my inbox is for active things that concern me, things that I need to stay on top of (including things where I need to do something and then send a reply).

(The problem with the side folders is that I never look at them, so they usually have plenty of now-concluded conversations that I could delete. But pruning them is the kind of low priority boring work that's hard to get around to.)

In general, what stays in my inbox is messages and conversations where I want to do something as a reaction to them; reply to them, read something they're pointing me to, keep track of whether a condition has cleared, and so on. While I get a constant flow of automated status messages, alerts, and so on, I read (or skim) these and immediately delete them unless they're actually something I need to react to. I try not to use my inbox as a todo list (I have a separate system for that), but I'm not always entirely successful.

(One reason not to use my inbox as a todo list is that my mail client only shows me a few of the most recent messages. Once a message disappears off that list I'm no longer being reminded of it and poof, it's gone from my mind.)

I don't have a good solution for things that I want to get around to but that I'm not. I don't always want to delete them and they shouldn't really stay in my inbox for long, but I'm always reluctant to refile them somewhere else because I know that makes it basically sure I'll never deal with them (and it's hard to give up on the idea that someday I will etc).

I should mention this explicitly: I don't tolerate unread messages in my inbox. I always read (or skim) everything right away, whether or not I do anything with it. To make this viable, I strongly limit what I let get into my inbox; almost everything that winds up there is at least theoretically important and relevant to me.

(I almost never subscribe to mailing lists (there are better alternatives) and I've ruthlessly pruned various web site notifications and so on down to things that I actually care about. Every so often I conduct another slash and burn round; for extreme cases I 'unsubscribe' by deleting the email alias that I gave wherever it was. As for automated status messages from local stuff, I procmail away as many of the 'everything is fine' status messages as I can and I try to make systems quiet in general.)

sysadmin/HowIUseEmail written at 01:33:06; Add Comment

Page tools: See As Normal.
Login: Password:
Atom Syndication: Recent Pages, Recent Comments.

This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.