== String expansion and securely running programs on Unix

One of the corollaries of [[how to securely run programs on Unix
SecurelyRunningPrograms]] is that a general purpose, generic string
expansion system is a bad fit with securely running programs. The
problem is that there is a fundamental clash of goals between the two
systems: a generic string expansion system wants to treat everything as
a generic string to be expanded (regardless of what it actually is), and
a secure system for running programs wants to tokenize everything using
simple rules.

At this point I am going to pick on Exim for illustrative
examples. Unfortunately, Exim tries to have it both ways at once and
thus is a great source for showing the problems that this causes, no
matter how much I like it otherwise. Please note that the problems here
are generic; any program that takes either approach (or both at once as
Exim does) will have the same issues.

First up is Exim's ((av_scanner)) setting. This is not expanded at all
unless it starts with a '$', at which point the entire string must
be expanded before Exim knows how to tokenize it:

> ((av_scanner = ${if bool{true} {cmdline:/opt/avscanner $recipients %s}}))

If you are concerned about arbitrary characters appearing in
_$recipients_, there is no way to make this secure ([[as discussed
before SecurelyRunningPrograms]]).

Second, the ((command)) setting for running things in pipes.  This
tokenizes things before string expansion, but it does the tokenization
purely on a textual basis. As the documentation notes, this causes
serious problems:

> ((command = /some/path ${if eq{$local_part}{postmaster} {xx} {yy}}))

Since tokenization is expansion-blind, this fails because all the string
expansion evaluator winds up seeing is '_${if_' (which is a clear syntax
error). To get this to work you have to force the tokenizer to treat the
entire string expansion as a single token by 'quoting' it.

(The documentation does not quite put it the way that I have here.)

A side effect of tokenization before expansion is that a single string
expansion can only ever expand to a single argument. (You may or may not
be able to expand to nothing instead of a _''_ empty argument, depending
on the implementation.)

What this points out is that ~~command line tokenization and string
expansion need to be aware of each other~~. Once the dust settles,
either string expansion needs to be able to mark hard token boundaries
(so that _$recipients_ can be marked as a single token regardless of
contents) or tokenization needs to know about the string expansion
language (so that ((${if ...})) can be parsed into a single token
despite the presence of internal spaces or [[other special characters
../sysadmin/ProgrammableMailersII]]).

(I have opinions on the answer here, but this entry is already long
enough as it is.)

PS: if you want to be secure with minimal effort, it's clear that you
need to do tokenization before expansion and provide some sort of
'quoting' mechanism to glue a string expansion expression into a single
token. This is secure while being merely inconvenient and annoying
to people writing configuration files. [[Simple expansion before
tokenization cannot be made secure at all SecurelyRunningPrograms]], as
previously discussed.