Changing major version numbers does not fix compatibility issues
So now that we know what's wrong, how do we fix it? The answer is to clean up the technical debt, to go through the code and make systematic changes to bring it up to 2014 standards.
This will fix a lot of bugs, but it will break existing shell-scripts that depend upon those bugs. That's not a problem -- that's what upping the major version number is for. [...]
There are several issues here, but let me start with the last statement. This statement is wrong, and let me say it plainly: changing your major version number does not fix compatibility problems. Changing your version number is not some magic get out of problems card that makes your new version compatible with your old version; regardless of its version number it remains just as incompatible with the old version as before. The purpose of changing version numbers is to warn people of potential problems, and by extension to reassure them that there won't be problems for minor version changes (which makes them more likely to adopt those changes).
If you introduce breaking changes in Bash or in any program, things that people have today will break. That is what breaking changes mean. People will still have to either fix them or stay with the old version of the program and yes, the latter really is an option whether or not you like it (it is especially an option if you are proposing changes in something that is not actually your program). When you introduce breaking changes you are asking people do extra work, which they do not enjoy in the least; generally the more breaking changes you introduce the more work you create. If you want your changes to be accepted, you need to be very sure that what you are offering is worth that work. Otherwise you are going to get an unpleasant surprise.
Changing the major version number does absolutely nothing to change this dynamic and thus it does nothing at all to make breaking changes less of a problem or less of an issue. You cannot hand wave away the cost of breaking changes this way, no matter how much programmers would love to do so and how this 'we'll just change major versions' thing is an ever-popular idea in languages, libraries, systems, commercial software, tools, and so on.
(Note that being left with no real option but to do a bunch of work also drastically lowers the cost of moving to some other alternative; you no longer have the difference between 'do nothing' and 'do the migration', now it is just the difference between 'fix the existing stuff' and 'do the migration'.)
DWiki, Python 3, Python, and me
A while back I tweeted:
Programming in #golang remains fun. I'm not sure if this is true for me for Python any more, but maybe I need the right project.
One of the problems for me with Python programming is that I kind of have a millstone and this millstone intersects badly with Python 3, which I kind of want to be using.
I have a number of Python projects, both work and personal. The stuff for work is not moving to Python 3, in significant part because most of our systems don't have good versions of Python 3 (or sometimes any versions of it). Most of my personal Python projects are inactive (eg), generally because I don't have much use for them any more. The personal project that is the exception is DWiki, the software behind Wandering Thoughts.
Unfortunately DWiki's source code is kind of a mess and as a result DWiki itself is sort of a millstone. DWiki has grown far larger than I initially imagined it ever would be and I didn't do a great job of designing it from the start (partly because I did not really understand many of the problems I was dealing with when I started writing it, which resulted in some core design missteps, and partly because it changed directions during development). The code has wound up being old and tangled and not very well commented. One of the consequences of this is that making any changes at all takes a surprising amount of work, partly just to recover my original understanding of the code, and as a result I need to have a lot of energy and enthusiasm to actually start trying to make any change.
(For instance, I've wanted to add entry tags to DWiki for a long time and I even have a strawman design in my head. What I haven't had so far is enough time and energy to propel me to dive into the code and get it done. And partly this is because the idea of working on the mess of DWiki's code just plain saps my enthusiasm.)
DWiki is currently a Python 2 program. I expect that moving it to Python 3 would take a fair amount of work and a lot of mucking around in the depths of its code (and then a bunch more work to make it use any relevant exciting Python 3 features). In fact the very idea of attempting the conversion is extremely daunting. But at the same time DWiki is the only Python program I'm likely to work on any time soon and the only one that is really important to carry forward to a Python 3 future (because it's the one program I expect to be running for a long time).
(Of course DWiki has no tests as such, especially unit tests. My approach for testing important changes is scripting to render all pages of CSpace in an old and a new code version and then compare the rendered output.)
So there my millstone sits, sapping my enthusiasm for dealing with it and by extension my enthusiasm for Python 3. I would be reasonably happy to magically have a Python 3 version of DWiki and I'm sure it would prompt me to dive into Python 3 in a fairly big way, but I can't see how I actually get to that future. Life be different if I could see a way that Python 3 would be a really big win for DWiki (such as significantly speeding it up or allowing me to drastically simplify chunks of code), but I don't believe that (and I know that Python 3 will bring complications).
(Life would also be different if DWiki didn't work very well for some reason (or needed maintenance) and I clearly needed to do something to it. But the truth is it works pretty well as it is. It's just missing wishlist items, such as tags and running under Python 3.)
PS: on the other hand, if I keep thinking and talking about DWiki and Python 3, maybe I'll talk myself into trying a conversion just to see how difficult it really is. The idea has a certain perverse attraction.
Sidebar: Why a major rewrite is not the answer
At this point some people will urge me to gut major portions of the current code (or all of it) and rebuild it from scratch, better and cleaner and so on. The simple answer about this is that if I was going to redo DWiki from more or less scratch (which has a number of attractions), I don't see why I'd do it in Python 3 instead of in Go. Programming in Python 3 would likely be at least somewhat faster than in Go but I don't think it would be a massive difference, while the Go version would almost certainly run faster and it would clearly have a much simpler deployment story.
So why not go ahead and rewrite DWiki in Go? Because I don't want to do that much work, especially since DWiki works today and I don't think I'd gain any really major wins from a rewrite (I've pretty much scrubbed away all of DWiki's pain points for day to day usage already).