Wandering Thoughts archives

On public areas on the Net and conversations therein

There have been essentially public areas on the 'net for a long time (from the time before the 'net was the Internet). In all of that time, a pattern that repeats over and over is that they get used for what I'll call closed discussions among an in crowd. These discussions happen in public (in a Usenet newsgroup, on a public mailing list or website, on IRC, on Twitter, etc) so they're not private, but they're not public in the usual sense because they're not open to outside participants to butt in on. When this closed nature is not supported by the technology of the medium (which it usually isn't), it will instead be supported by social mores and practices, including ignoring people and the equivalent of mail filters and Usenet killfiles. There may be flaming or mockery of transgressors involved, too.

(If you want an analogy, what is going on is much like a group of people having a discussion at a restaurant table with you one table over. You can hear them fine and they are in 'public', but of course very few people think it's correct to turn around and join in their discussion and doing so rarely gets a good reaction from the group.)

What this means is that a conversation taking place in nominal public is not necessarily an open invitation for outside people to comment, and if they do they may be summarily ignored or find that there are bad reactions to their words. Equally, it's wrong to assert something like 'all conversations in public must include anyone who wants to participate' or the equivalent, because this is not how things work in practice in the real world (either on the 'net or off it).

As I mentioned, people on the 'net have been doing this with public spaces for a very long time now; this behavior is not at all novel or unusual. People who are shocked, shocked to see this happening in any particular instance (especially when they are shoving themselves into other people's discussions) are at best disingenuous. Wherever they are, people make groups and then talk among themselves.

There are also genuinely open public discussions in those public areas of the 'net, which creates obvious possibilities for confusion and misunderstandings. The cues for closed discussions are not always clear and some number of closed discussions are in practice only semi-closed; if you fit in, you can join in the conversation (indeed, this is how many such discussion groups expand). One way to assess the line between good faith misunderstandings of a situation and something else is the degree of stubborn persistence exhibited by the outsider.

(Relevant, and also.)

The problems with creating a new template language

One reaction to my entry saying you shouldn't create new templating languages is to ask why this is so. My original entry was written from the perspective of someone who's actually done this so I just assumed that all of the problems with creating your own were obvious, but this is not necessarily the case. So let's run down the problems here.

When you create a new web templating system, you face a number of problems:

• You need to design the templating language. Language design is hard and my own experience strongly suggests that you don't want a (too) minimal design. Many of the design decisions you make here will constrain your further steps and what can be done with the templating system, often in ways that are not necessarily obvious until later.

  (A too complicated templating language has its own drawbacks as well, but there are tradeoffs and decisions that depend on the environment that the templating system will be used in. For example, are the people writing the templates also going to be coding the web app, so that you can move complexity from the templating language to the app itself, or are different groups doing each side of things?)

• You need to design the API for using the templating system; how you specify and load templates, how you expand them, how you provide data used during expansion, and so on. As part of this you will face issues of what happens in various sorts of errors. If your template language has loops or other potentially unbound constructs, you're going to need to decide how you limit them (or if you just let template coding errors cause template expansion to run forever).

  One issue that you will want to consider is how expansion strings will or won't be automatically escaped by the templating system. Not doing escaping at all has proven to be extremely dangerous, but at the same time inserting unescaped text is sometimes necessary. HTML has several different contexts that need different sorts of things escaped, then there's URLs, and you may also want your templating system to be useful for more than HTML.

• You need to actually write all of the code. I hope that by now you see that this is much more sophisticated than just printf expansion of strings; we're talking about a full scale parser and interpreter of your language, which probably has conditions, loops, and so on. In the process of this (if you haven't already done so earlier), you're going to wind up dealing with character set conversion issues.

• Once you've written the basic template handler there are a bunch of efficiency issues that come into the picture. A good template system does not reparse everything from scratch on every request, which means both pre-parsing things and figuring out how to detect when you need to reload changed template files off disk. Then there's various sorts of caching for template expansion, and perhaps you want some way to generate ETag and Last-Modified information without running a full template expansion (and then often throwing away the result). Can you write out partial template results in chunks to hold down memory requirements, or do you have to fully generate even huge pages in memory before you can start output?

  (And there's the efficiency issue of simply making the code run fast. Profiling and performance tuning code takes work all by itself.)

You can write simpler templating systems that skip some or many of these considerations. Some of them are relatively unimportant at small scale (DWiki gets by without any sort of template preparsing) but others may cause you serious security problems if you neglect them. On top of that, there are any number of issues that have proven to be inobvious to people who are writing their first templating system. No matter what scale of templating system you're writing, you can expect to run into problems that you don't even initially recognize or realize that you have.

(I'm not even convinced I know how to design and write a good templating system, and I have the advantage of having done it once already.)

Using an existing templating system instead of writing your own has the great advantage that other people have already worried about and faced all of these issues. If you pick a good templating system, other people should have already invested all of the time and work to come up with good solutions (and they will continue to put effort into things like bug fixes and performance improvements). In fact they may well have solved problems you don't yet realize even exist.

All of that work saving is nice. But there's a deeper reason not to roll your own here:

You are probably not going to do as good a job as existing template systems do.

Writing a good templating system is hard work that takes a lot of specialized knowledge and skill. Unless you put a quite large amount of time into it, your new templating system is very likely to not be as nice as existing templating system. It will be incomplete and inefficient and limited and possibly buggy and problematic. This should not be surprising, since major templating systems have had a great deal of work put into them by a bunch of smart people. It would be really amazing if you could duplicate that all on your own in a relatively small amount of time.

(Of course you may have the advantage of writing a more focused and narrower templating system than those major templating systems, which tend to be quite general. My personal opinion is that you're probably not going to be making one that's narrow enough to make up all that ground.)