Some notes on providing Python code as a command line argument

May 24, 2022

I've long known about CPython's '-c' argument, which (in the words of the manual page) lets you "specify the command [for CPython] to execute". Until recently, I thought it had to be a single statement, or at least a single line of Python code (which precluded a number of things). It turns out that this isn't the case; both CPython and PyPy will accept a command line argument for -c that contains embedded newlines, in the style of providing command line code to Unix tools like awk.

For example:

python -c 'import sys
if len(sys.argv) > 1:
   print("arguments:", sys.argv[1:])
   print("no arguments")' "$@"

(For various reasons, you still might want to make this code importable, although I haven't done so here.)

If you're directly supplying the code on the command line, as I am here, you have a choice (in a Bourne shell script or environment). You can quote the entire code with single quotes and not use a literal single quote in the Python code, or you can quote with double quotes and carefully escape several special characters but get to use single quotes. If you want to avoid all of this, you need to put the code into a shell variable:

pyprog="$(cat <<'EOF'
python -c "$pyprog" ...

As you'd expect, '__name__' in the command line code is the usual '__main__'. As the manual page covers, all further command line arguments as passed in sys.argv, with sys.argv[0] set to '-c'. Since the code doesn't have a file name (which is what would normally go in sys.argv[0]), this seems like a decent choice, and immediately passing further arguments to the code is convenient.

Although this makes it possible to have a Python program embedded into a shell script in the same way that you can do this with awk (and thus implicitly helps enable Python as a filter in a shell script), I personally don't find the idea too appealing, at least for Python code of any substance. The problem isn't the need to take extra care with embedding the Python code in your shell script, although that's not great. The real problem is that embedding Python code this way means you miss out on all sorts of tools that are in the Python programming ecology, because they only work on separate Python code.

(If I had to write something this way, I would be tempted to develop it in a separate file that the shell script invoked with 'python <filename>' instead of 'python -c', and then only embed the code into the shell script and switch to 'python -c ...' at the last moment.)

PS: Now that I know how to do this it's a little bit tempting to try out small amounts of Python code in places where awk doesn't quite have the functions and power I'd like (or at least doesn't make the functions as easy as Python does). On the other hand, awk doesn't make you think about character set conversion issues. Probably I wouldn't use this to parse and reformat smartctl's JSON, though. That's likely to be enough code that I'd want to use the usual Python tools on it.

Comments on this page:

By Michael at 2022-05-25 02:56:36:

Another option might be to use a here-document for the code itself, thus embedding it into the shell script file, but to instead output the embedded script to a temporary file and point the interpreter at that file for execution.

Something like:

   cat >"$pyfile" <<'EOF'
   python "$pyfile" ...
   rm "$pyfile"

(You might want some error handling, signal catching and so on to make sure the temporary file gets cleaned up on an unclean exit from the script, or you could just rely on whatever other functionality you have in place to remove stale temporary files.)

Presumably, this would allow you to do anything within the embedded script that can be done with a standalone script written in the same language, without the hassle of having to ship it as a separate file; the script itself takes care of that part. The downside, of course, is that you still have all the downsides of embedding, such as having to be very careful with shell-significant characters such as $.

For Python specifically, a quick glance at the man page suggests that the interpreter also accepts - as the source file, in which case I guess you could pipe the output of cat directly into python - (or maybe elide cat entirely and just provide the here-document to python) to execute it without storing it anywhere else first. I don't know how that compares to either other option.

Written on 24 May 2022.
« Systems should expose a (simple) overall health metric as well as specifics
Shell scripts should be written to be clear first »

Page tools: View Source, View Normal, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Tue May 24 22:31:44 2022
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.