The problems with creating a new template language

November 16, 2015

One reaction to my entry saying you shouldn't create new templating languages is to ask why this is so. My original entry was written from the perspective of someone who's actually done this so I just assumed that all of the problems with creating your own were obvious, but this is not necessarily the case. So let's run down the problems here.

When you create a new web templating system, you face a number of problems:

  • You need to design the templating language. Language design is hard and my own experience strongly suggests that you don't want a (too) minimal design. Many of the design decisions you make here will constrain your further steps and what can be done with the templating system, often in ways that are not necessarily obvious until later.

    (A too complicated templating language has its own drawbacks as well, but there are tradeoffs and decisions that depend on the environment that the templating system will be used in. For example, are the people writing the templates also going to be coding the web app, so that you can move complexity from the templating language to the app itself, or are different groups doing each side of things?)

  • You need to design the API for using the templating system; how you specify and load templates, how you expand them, how you provide data used during expansion, and so on. As part of this you will face issues of what happens in various sorts of errors. If your template language has loops or other potentially unbound constructs, you're going to need to decide how you limit them (or if you just let template coding errors cause template expansion to run forever).

    One issue that you will want to consider is how expansion strings will or won't be automatically escaped by the templating system. Not doing escaping at all has proven to be extremely dangerous, but at the same time inserting unescaped text is sometimes necessary. HTML has several different contexts that need different sorts of things escaped, then there's URLs, and you may also want your templating system to be useful for more than HTML.

  • You need to actually write all of the code. I hope that by now you see that this is much more sophisticated than just printf expansion of strings; we're talking about a full scale parser and interpreter of your language, which probably has conditions, loops, and so on. In the process of this (if you haven't already done so earlier), you're going to wind up dealing with character set conversion issues.

  • Once you've written the basic template handler there are a bunch of efficiency issues that come into the picture. A good template system does not reparse everything from scratch on every request, which means both pre-parsing things and figuring out how to detect when you need to reload changed template files off disk. Then there's various sorts of caching for template expansion, and perhaps you want some way to generate ETag and Last-Modified information without running a full template expansion (and then often throwing away the result). Can you write out partial template results in chunks to hold down memory requirements, or do you have to fully generate even huge pages in memory before you can start output?

    (And there's the efficiency issue of simply making the code run fast. Profiling and performance tuning code takes work all by itself.)

You can write simpler templating systems that skip some or many of these considerations. Some of them are relatively unimportant at small scale (DWiki gets by without any sort of template preparsing) but others may cause you serious security problems if you neglect them. On top of that, there are any number of issues that have proven to be inobvious to people who are writing their first templating system. No matter what scale of templating system you're writing, you can expect to run into problems that you don't even initially recognize or realize that you have.

(I'm not even convinced I know how to design and write a good templating system, and I have the advantage of having done it once already.)

Using an existing templating system instead of writing your own has the great advantage that other people have already worried about and faced all of these issues. If you pick a good templating system, other people should have already invested all of the time and work to come up with good solutions (and they will continue to put effort into things like bug fixes and performance improvements). In fact they may well have solved problems you don't yet realize even exist.

All of that work saving is nice. But there's a deeper reason not to roll your own here:

You are probably not going to do as good a job as existing template systems do.

Writing a good templating system is hard work that takes a lot of specialized knowledge and skill. Unless you put a quite large amount of time into it, your new templating system is very likely to not be as nice as existing templating system. It will be incomplete and inefficient and limited and possibly buggy and problematic. This should not be surprising, since major templating systems have had a great deal of work put into them by a bunch of smart people. It would be really amazing if you could duplicate that all on your own in a relatively small amount of time.

(Of course you may have the advantage of writing a more focused and narrower templating system than those major templating systems, which tend to be quite general. My personal opinion is that you're probably not going to be making one that's narrow enough to make up all that ground.)

Comments on this page:

By Ewen McNeill at 2015-11-16 03:14:05:

There's another disadvantage in creating your own templating system: either you use the exact syntax of an existing templating system (in which case why are you implementing it again?) or you use a different syntax in which case you're rapidly heading towards xkcd #927. This is particularly so if your templating system is going to end up being one of a kind, and even more so if it's similar-but-different to something common.

I'd make a partial exception for "same template syntax, different implementation language" -- but probably only in new languages without a well established templating system -- and for strict subsets of an existing template syntax. Because a strict subset at least reduces the cost of making new a new "standard" -- users just have to remember which parts are implemented -- and if you discover that it's all getting too complicated, you can at least still bolt in the existing template system instead, without having to rewrite your markup.


By cks at 2015-11-16 10:37:40:

You implicitly point out another problem that I forgot to list: when you create a new system, you need to document it, at least as far as covering the template language and the API.

Reusing an existing template language and API reduces some of the risks (and if you're working in a new language with no decent templating system, I do think that you should do that). But it still leaves you to write the code and wrestle with all of the internal issues.

By Karl Vogel, at 2015-11-21 22:05:30:

Minor nit (if you made this point and I missed it, I need new glasses): it's great exercise to write your own template system as long as you either don't use it, or write something else that can convert stuff using it to a known and supported template language.

I tried writing my own system a long time ago, and after the first few hideous botches (including a dandy example of second-system-syndrome), I stopped complaining about other supported systems; I'd run into similar problems and my bright ideas for fixing them were never any better.

Trying to write your own can also make it easier to poke around in the guts of someone else's system because you can recognize the same basic areas, like what happens when you don't follow the syntax rules? Does the template handler barf all over the screen? Lose your input, if it's interactive? Give something resembling a useful error message?

I realize this is an old entry (you have a lot of back catalog here, that I wander around randomly) but this is hilarious. I think it was about the third time through writing my own template system that I realized these problems; I wrote about my first two designs on my blog for everyone to facepalm over.

The third system died when I eventually needed to add yet another feature, and I could see another potential feature on the horizon. Implementing them both right away would be a lot of work, and it would be even more work to implement them at separate times. That finally made me switch everything to Twig, and extend it so that email templates could export Subject.

It's also funny that over time, my designs converged on standard template systems. It turns out that the common approach is common because it sits at a good place among all the tradeoffs.

Written on 16 November 2015.
« I should remember that I can cast things in Go
On public areas on the Net and conversations therein »

Page tools: View Source, View Normal, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Mon Nov 16 01:34:32 2015
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.