The problems with creating a new template language

November 16, 2015

One reaction to my entry saying you shouldn't create new templating languages is to ask why this is so. My original entry was written from the perspective of someone who's actually done this so I just assumed that all of the problems with creating your own were obvious, but this is not necessarily the case. So let's run down the problems here.

When you create a new web templating system, you face a number of problems:

  • You need to design the templating language. Language design is hard and my own experience strongly suggests that you don't want a (too) minimal design. Many of the design decisions you make here will constrain your further steps and what can be done with the templating system, often in ways that are not necessarily obvious until later.

    (A too complicated templating language has its own drawbacks as well, but there are tradeoffs and decisions that depend on the environment that the templating system will be used in. For example, are the people writing the templates also going to be coding the web app, so that you can move complexity from the templating language to the app itself, or are different groups doing each side of things?)

  • You need to design the API for using the templating system; how you specify and load templates, how you expand them, how you provide data used during expansion, and so on. As part of this you will face issues of what happens in various sorts of errors. If your template language has loops or other potentially unbound constructs, you're going to need to decide how you limit them (or if you just let template coding errors cause template expansion to run forever).

    One issue that you will want to consider is how expansion strings will or won't be automatically escaped by the templating system. Not doing escaping at all has proven to be extremely dangerous, but at the same time inserting unescaped text is sometimes necessary. HTML has several different contexts that need different sorts of things escaped, then there's URLs, and you may also want your templating system to be useful for more than HTML.

  • You need to actually write all of the code. I hope that by now you see that this is much more sophisticated than just printf expansion of strings; we're talking about a full scale parser and interpreter of your language, which probably has conditions, loops, and so on. In the process of this (if you haven't already done so earlier), you're going to wind up dealing with character set conversion issues.

  • Once you've written the basic template handler there are a bunch of efficiency issues that come into the picture. A good template system does not reparse everything from scratch on every request, which means both pre-parsing things and figuring out how to detect when you need to reload changed template files off disk. Then there's various sorts of caching for template expansion, and perhaps you want some way to generate ETag and Last-Modified information without running a full template expansion (and then often throwing away the result). Can you write out partial template results in chunks to hold down memory requirements, or do you have to fully generate even huge pages in memory before you can start output?

    (And there's the efficiency issue of simply making the code run fast. Profiling and performance tuning code takes work all by itself.)

You can write simpler templating systems that skip some or many of these considerations. Some of them are relatively unimportant at small scale (DWiki gets by without any sort of template preparsing) but others may cause you serious security problems if you neglect them. On top of that, there are any number of issues that have proven to be inobvious to people who are writing their first templating system. No matter what scale of templating system you're writing, you can expect to run into problems that you don't even initially recognize or realize that you have.

(I'm not even convinced I know how to design and write a good templating system, and I have the advantage of having done it once already.)

Using an existing templating system instead of writing your own has the great advantage that other people have already worried about and faced all of these issues. If you pick a good templating system, other people should have already invested all of the time and work to come up with good solutions (and they will continue to put effort into things like bug fixes and performance improvements). In fact they may well have solved problems you don't yet realize even exist.

All of that work saving is nice. But there's a deeper reason not to roll your own here:

You are probably not going to do as good a job as existing template systems do.

Writing a good templating system is hard work that takes a lot of specialized knowledge and skill. Unless you put a quite large amount of time into it, your new templating system is very likely to not be as nice as existing templating system. It will be incomplete and inefficient and limited and possibly buggy and problematic. This should not be surprising, since major templating systems have had a great deal of work put into them by a bunch of smart people. It would be really amazing if you could duplicate that all on your own in a relatively small amount of time.

(Of course you may have the advantage of writing a more focused and narrower templating system than those major templating systems, which tend to be quite general. My personal opinion is that you're probably not going to be making one that's narrow enough to make up all that ground.)

Written on 16 November 2015.
« I should remember that I can cast things in Go
On public areas on the Net and conversations therein »

Page tools: View Source, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Mon Nov 16 01:34:32 2015
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.