2013-12-31
Two uses of fmt
The venerable fmt program is not something that I normally think of as
a command that I use in my scripts or on the fly pipelines; I usually
think of it more as, say, something that I use to reflow paragraphs in
vi. But it has quietly been making its way into an increasing number
of them because it turns out that fmt is the easy and lazy way to do
two symmetrical things: to word-split lines and to merge 'one word per
line' output back to a single line.
Word splitting is straightforward:
somecmd | fmt -1
Every version of fmt that I've seen accepts this and does the right
thing; you get one word per line after fmt.
Word joining is much more annoying and it's all because of GNU
coreutils. Real versions of fmt accept very large format widths:
/bin/ls -1 | fmt -9999
Unfortunately the GNU coreutils version of fmt has a maximum width
of 2500 characters. Worse, it has a 'goal' width that defaults to 93%
of the actual width, so if you're worried about getting close to that
limit you need to use:
/bin/ls -1 | fmt -2500 -g2500
In practice I usually use 'fmt -999' in command pipelines because my
actual output line is going to be nowhere near 999 characters to start
with.
(Usually when I'm doing word merging it's because I'm going to take the one line that results and paste it to something else, which imposes a relatively modest line length limit in practice.)
What this points out is that fmt is not really the ideal solution
to this (and in fact the FreeBSD version of fmt also has oddities,
such the man page's
description of behavior around the -n switch). The traditional Unix
solution to these problems is tr, using it to either turn spaces to
newlines or newlines to spaces. The problem for me in practice is that
to use tr I need to remember or re-derive the octal value of newline
(it's \012, by the way) and that is just a bit too much hassle. So I
use fmt, vague warts and all.
(The other drawback of tr is that 'tr " " "\012"' will have a
trailing space and no final newline. Usually this is not a big deal.)
Actually in writing this I've discovered that I'm behind the times. All
of the versions of tr that I use today will accept \n instead of
the octal literal. Either there was a day when this wasn't true or I
just never read far enough in the tr manpage (and had it stick) to
notice that particular feature. (I'm probably still going to keep on
using fmt, though.)
Reversing my view on Python 3 for new general code: avoid it
Just about exactly two years ago I wrote Python3NewCode, in which I waved my hands a bunch and then said:
Ignoring [my handwaved issues] as ultimately unimportant, I don't think there's any reason not to write new, non-sysadmin code in Python 3.
I take all of that back. In retrospect I was being too nice to Python 3 back then and I was wrong to do so. Here is my new view: you should avoid Python 3 even for new code because there is very little to gain from writing in Python 3 and significant downsides to doing so.
(Part of those downsides is that the things that I so blithely handwaved away did not in fact go away and remain as real issues today, two years after I wrote that entry.)
The spark for this reassessment is twofold. First, I have not actually written any of my new Python code in Python 3 (for all sorts of reasons I'm not going to belabor). Second, Alex Gaynor recently wrote 'About Python 3' and this got me thinking about the whole situation and my feelings.
The big problem with Python 3, per Gaynor's article, is that the Python 3 ecosystem is a ghost town. Regardless of whether or not you have Python 3 available on any particular system, the reality is that almost no one is writing Python 3 code. The practical Python ecosystem, the one where people will answer your questions and develop interesting new modules and interesting Python things is Python 2. Useful deployment options are in practice Python 2 ones. If you choose to write in Python 2, you get to take advantage of all of this. If you write in Python 3, not so much. In exchange for giving up all of this you get very little. Most people will find no killer, must-have feature in Python 3 to compensate for the risks and problems you are taking on by using it.
(There are some modules that are only available for Python 3. My impression is that for the most part they come from the core Python developers, precisely because all outside people who are developing modules understand that most of the actual Python programming happens in Python 2.)
Given the complete shambles of the Python 2 to Python 3 transition and the resulting uncertainty about what's going to happen in the longer term, I can't recommend starting even greenfield development in Python 3 unless you have some quite strong reason for it (ie, something important that you can do in Python 3 but not in Python 2). Certainly I reverse my previous position; there's no strong reason to write new code in Python 3 and some good reasons to avoid doing so. Python 2 is here and good today. Even today, Python 3 is still a 'someday maybe in the future' thing.
(At this point I'm not sure if a genuine Python 2 to Python 3 transition will ever happen. The really pessimistic future is that Python 2 shambles on as an increasingly creaky zombie for the next decade, Python 3 effectively fails and becomes irrelevant, and as a result Python itself is abandoned for other languages.)
Link: Alex Gaynor's 'About Python 3'
Alex Gaynor just wrote About Python 3, which is not a bright and happy assessment about the state of Python 3. He says many things that I agree wholeheartedly with, from a position of authority and of good writing. He also crystallizes a number of things for me, such as the following:
Since the time of the Python 3.1 it's been regularly said that the new features and additions the standard library would act as carrots to motivate people to upgrade. Don't get me wrong, Python 3.3 has some really cool stuff in it. But 99% of everybody can't actually use it, so when we tell them "that's better in Python 3", we're really telling them "Fuck You", because nothing is getting fixed for them.
Yes. This. Wholeheartedly this. Every Python 3 only feature or module or improvement might as well be on the far side of the moon as far as it goes for me using it for anything meaningful.
And what he says at the end, too. Everything that the core Python developers are currently doing is completely irrelevant to what I do with Python and will probably be for at least five more years and perhaps as much as a decade. At this point we are living on different planets.
By the way, significant problems surfacing with Python 2 and not getting fixed would not get me to migrate to Python 3. I cannot migrate to Python 3 at this point because it is simply not present on platforms that I use. Very soon my best alternative to Python 2 will probably be Go, because at least I'll be able to compile static binaries for just about everything I care about and push them to the target machines.
(Using Go will suck compared to using Python for the problems that I use Python for, but it will suck less than building and installing my own version of Python 3.)
This is a drum that I have been banging on for some time so of course I'm happy to see it getting attention from people with influence, instead of just my lone voice in a corner. I'd like to think that people like Alex Gaynor speaking up will create actual change but I don't expect that to happen at this point. The core Python developers have to be very invested in their vision of Python 3 and its transition by now; a significant reversal would be very surprising because people almost never reverse such deeply held positions regardless of what they are.