Python's os.environ is surprisingly liberal in some ways

February 27, 2022

The way you access and modify Unix environment variables in Python programs is generally through os.environ; Python 3 being Python 3, sometimes you need os.environb. In Unix, what can go in the environment is somewhat fuzzy and while Python has some issues with character encodings, it's otherwise surprisingly liberal in a number of ways.

The first way that os.environ is liberal is that it allows environment variables to have blank values:

>>> os.environ["FRED"] = ""
>>> subprocess.run("printenv")
[...]
FRED=
[...]

It's possible to do this with some Unix shells as well, but traditionally environment variables are generally assumed to have non-blank values. Quite a lot of code is likely to assume that a blank value is the same as the variable being unset, although in Python you can tell the difference since os.environ raises KeyError if the environment variable doesn't exist at all.

A bigger way that os.environ is liberal is that it will allow you to use non-traditional characters in the names of environment variables:

>>> os.environ["FRED/BAR"] = "Yes"
>>> subprocess.run("printenv")
[...]
FRED/BAR=Yes

On Unix, setting an environment variable uses setenv(), which generally only requires that you avoid '='. Python specifically checks for an '=' in your name so that it can generate a specific error, and otherwise passes things through.

Python itself doesn't particularly restricted environment variable names beyond that. As a result you can do all sorts of odd things with environment variable names, including putting spaces and Unicode into them (at least in a UTF-8 environment). Some or many of these environment variables won't be accessible to a shell program, but not everything that interprets the environment follows the shell's rules.

The case where this came up for me recently was in Dovecot post-login scripting, which in some cases can require you to create environment variables with '/' in their names. Typical shells disallow this, but I was quite happy to find that Python was perfectly willing to go ahead and everything worked fine.


Comments on this page:

The shell sort of takes both attitudes on the question of unset vs empty variables, making the difference very visible on one hand while on the other hand also making it really easy to ignore. As one (prominent) example, parameter expansion syntax not only allows distinguishing the cases but even treats unset variables as the primary case: you need to add an extra character to the operator to treat empty variables the same way. On the other hand, the very fact of how slight this difference is is telling. In practice I have never seen anyone use an expansion operator without the colon added, and I suspect a lot of people who write shell scripts don’t even realise that the colon isn’t just a fixed part of the syntax.

Given that the shell is the 800-pound gorilla when it comes to dealing with environment variables, its both-ways stance forces everyone else into the same “the difference between the cases doesn’t matter, except when it does” situation. The end result is that the difference is not useful in practice, it just has to be dealt with every once in a while, i.e. useless friction. It would have been nice if someone had picked one way or the other, back when that could have been done, and enforced it. But it’s also not remotely big enough of a problem to merit fixing now.

By Opk at 2022-03-03 19:07:17:

If you need weirdly named environment variables from the shell, you might find the external env command to be more obliging. A lot of tightened restrictions on variable names was added to a variety of shells in the aftermath of shell shock. The environment is a source of data that can't now be trusted from a security perspective.

Written on 27 February 2022.
« Unix environment variables (and the environment) are a fuzzy thing
Firefox (Nightly) and the case of the fading scrollbars on Unix »

Page tools: View Source, View Normal, Add Comment.
Search:
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Sun Feb 27 23:24:29 2022
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.