My view of Debian's behavior on package upgrades with new dependencies
Why is this [apt-get's
--with-new-pkgs] not enabled by default in debian / ubuntu ?
The ultimate answer here is 'because Debian has made a philosophical
choice'. Specifically, Debian has decided that no matter what the
person building the new version of a Debian package wants or feels
is necessary, an '
apt-get upgrade' will never add additional
packages to your system. If the builder of the package insists that
a new version requires an additional package to be installed, it is
better for the upgrade to not happen. Only '
apt-get install <pkg>'
apt-get dist-upgrade') will ever add new packages to your
Regardless of what you think about its merits, this is a coherent position for Debian to take. In an anarchic packaging environment with individual Debian developers going their own way, it even has a fair amount of appeal. It certainly means that package maintainers have a strong pragmatic incentive not to add new dependencies, which probably serves to minimize it (which is one reason Debian has apt-get behave this way).
My personal view is that I prefer an environment where package builders are trusted to do the right thing with package dependencies in new versions of their packages, whatever that is. Packages can get new dependencies for all sorts of good reasons, including that what used to be a single package is being split up into several ones. As a sysadmin outsider, I'm not in a good position to second guess the package maintainer on what dependencies are right and whether or not a new one is harmful to my system, so in a trustworthy environment I'll just auto-install new dependencies (as we now do on Ubuntu where it's possible).
(The Debian package format has also made some structural decisions that make things like splitting packages more of a pain. In an RPM-based system, other packages often don't notice or care if you split yours up; in a Debian one, they're more likely to notice.)
It's worth pointing out that this trust fundamentally requires work and politics, in that it requires a policy on 'no unneeded dependencies' (and 'no surprises in package upgrades') and then a group of people who are empowered to judge and enforce the policy (overriding package maintainers when necessary). This sort of control probably does not go well with a relatively anarchic project and it's certainly a point of argument (and one could say that Debian already has enough of those).