Wandering Thoughts archives


Another way that generators are not lists: modifying them

A long time ago, I wrote some stuff on how generators are not lists (okay, technically it was about iterators), and one of the things that I mentioned is that generators do not have list methods. Well, there's a consequence of that that only struck me recently: you need completely different code to modify a returned generator than to modify a returned list.

Suppose you have a function that returns something that is conceptually a list of items. Further suppose that you have another function that modifies what the first function returns; perhaps you want to add something on the end. If you know you're dealing with a list, you write:

def append(func, extra):
    r = func()
    return r

If func() is a generator, this code blows up. You have two choices; first, you can forcefully turn the result of func() into a list, and second, you can rewrite append() as a generator (which will work regardless of what func() returns, but may have consequences that make it undesirable):

def append(func, extra):
   for it in (func(), extra):
       for e in it:
           yield e

(Yes, yes, one can write this using itertools.chain(). Then people would have to look it up.)

In either case, you have to actively make a decision about what your function will do. You cannot passively modify whatever you get handed and pass it up to your caller without changing its nature; you must decide that no matter what func() returns, you're either returning a list or an iterator.

(Technically you can, since you can see if you got handed something that follows the iterator protocol or whether it looks sequence-like. But that way lies madness.)

python/GeneratorListModification written at 23:15:13; Add Comment

Monitoring systems should be always be usefully informative

There is a fashion in monitoring systems that, once you have one, you start monitoring and alerting on everything that you possibly can think of, no matter what it is. If you can measure it, you do, and when it gets too big or too small your system lets people know about it.

This is, by and large, a mistake.

It is a mistake because you've created a system that isn't actually (usefully) informative, just noisy. What your monitoring system should be telling you about is real problems that you can do something about, not things that either aren't real problems or aren't problems that you can do anything about.

(Take, for example, the ever popular monitoring of free user disk space. At least around here, there is nothing we can do it if a user filesystem runs out of space; we can neither go in and remove user files to get space back nor add more space that the users haven't paid for.)

The less noise your monitoring system has, the more likely people are to look at it and actually pay attention if it has trouble indicators. A monitoring system that always shows trouble indicators is about as useful as a fire alarm that is on all the time (although probably less annoying).

Yes, yes, people can learn to ignore 'known harmless' trouble indicators. The problem is that this takes mental work, which means that it takes more effort to check the monitoring system, which means people do it less often or pay less attention to it or both. It also means that you cannot look at a top level summary and get anything useful from it, because the overall system is never in 'all green' condition. And having something that you can quickly glance at to look for problems is a significant win.

Sidebar: the case for widespread monitoring

There is a case for tracking everything you can, provided that your monitoring system keeps history and can display 'measure over time' graphs or the like. Then what you're doing is getting statistics, which is vital. But if you're tracking things for statistics, you should not alert on them.

So by all means track user disk space usage, so that you can draw nice graphs about six month trends in space growth that clearly justify another shelf of disks. Just don't alert on them.

(This is one area where canned monitoring systems are your friends, because they have probably already got systems to keep lots of history of random measurements and graph them for you.)

sysadmin/InformativeMonitoring written at 01:45:17; Add Comment

Page tools: See As Normal.
Login: Password:
Atom Syndication: Recent Pages, Recent Comments.

This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.