Scrolling versus panning
In thinking more about my dislike of smooth scrolling, I think it's useful to distinguish scrolling through something from panning through it. The difference is that panning operations are a continuous process, whereas scrolling is a discrete thing that theoretically has a specific size that it's supposed to jump.
Since panning is continuous anyways, panning operations clearly can smooth 'scroll' without problems; it matches how people think of them to start with. Scrolling, especially scrolling large amounts, is not like that so (at least to me) using smooth scrolling feels wrong.
This still leaves you to figure out which operations users think of as panning ones and which ones they consider scrolling ones. Sometimes this is clear (my rule of thumb is that any continuous action is panning), but sometimes it is not; for example, should the mouse scrollwheel be considered panning or scrolling? In theory the scroll wheel is generally equivalent to cursor up and cursor down, which are discrete things (generally moving one line); however, I think most people use it in a more continuous manner and will expect it to smooth scroll.
(This is certainly how I find myself using a scrollwheel mouse; the scroll wheel is less for moving to a specific place in a document than for panning slowly through it.)
If you have well implemented smooth scroll you may want to use it even for small scrolling movements, such as cursor up/down. Beyond that, please don't, or at least have an option to not do it.