Linux is terrible at handling IO to USB drives on my machine

January 16, 2017

Normally I don't do much with USB disks on my machine, either flash drives or regular hard drives. When I do, it's mostly to do bulk read or write things such as blanking a disk or writing an installer image to a flash drive, and I've learned the hard way to force direct IO through dd when I'm doing this kind of thing. Today, for reasons beyond the scope of this entry, I was copying a directory of files to a USB flash drive, using USB 3.0 for once.

This simple operation absolutely murdered the responsiveness of my machine. Even things as simple as moving windows around could stutter (and fvwm doesn't exactly do elaborate things for that), never mind doing anything like navigating somewhere in a browser or scrolling the window of my Twitter client. It wasn't CPU load, because ssh sessions to remote machines were perfectly responsive; instead it seemed that anything that might vaguely come near doing filesystem IO was extensively delayed.

(As usual, ionice was ineffective. I'm not really surprised, since the last time I looked it didn't do anything for software RAID arrays.)

While hitting my local filesystems with a heavy IO load will slow other things down, it doesn't do it to this extent, and I wasn't doing anything particularly IO-heavy in the first place (especially since the USB flash drive was not going particularly fast). I also tried out copying a few (big) files by hand with dd so I could force oflag=direct, and that was significantly better, so I'm pretty confident that it was the USB IO specifically that was the problem.

I don't know what the Linux kernel is doing here to gum up its works so much, and I don't know if it's general or specific to my hardware, but it's been like this for years and I wish it would get better. Right now I'm not feeling very optimistic about the prospects of a USB 3.0 external drive helping solve things like my home backup headaches.

(I took a look with vmstat to see if I could spot something like a high amount of CPU time in interrupt handlers, but as far as I could see the kernel was just sitting around waiting for IO all the time.)

PS: We have more modern Linux machines with USB 3.0 ports at work, so I suppose I should do some tests with one just to see. If this Linux failure is specific to my hardware, it adds some more momentum for a hardware upgrade (cf).

(This elaborates on some tweets of mine.)


Comments on this page:

By Diego at 2017-01-16 08:58:30:

This has always been a problem as far as I can remember. I think that the main reason is that Linux memory management seems to be optimized for server benchmarks. Modeling "desktop responsiveness" in a benchmark is more boring than getting higher raw op/s.

By Anon at 2017-01-17 01:28:43:

The hope is that one day new kernels will throttle writeback I/O in a such a way that system remains responsive without the user having to fiddle around (but people have been complaining/asking for this for years).

Jens Axboe made the wb-buf-throttle patch series that tries to tackle this - see https://lwn.net/Articles/682582/ (https://lwn.net/Articles/681763/ for the description provided within the patches) which has been merged into the yet-to-be-release 4.10 kernel (https://lwn.net/Articles/708476/ ). What makes this interesting it that it should still work with extreme devices using blk-mq which doesn't have ioscheduler.

A different approach for devices that still have an ioscheduler is a better ioscheduler. The BFQ ioscheduler patches (still out of tree but there have been attempts - https://lwn.net/Articles/674308/ ) on http://algo.ing.unimo.it/people/paolo/disk_sched/ claim to dramatically improve latency under load even on SSDs. There's are some pleasingly easy to watch videos comparing it to the other schedulers using hard disks (https://youtu.be/ZeNbS0rzpoY ) and SSDs (http://youtu.be/1cjZeaCXIyM ).

Written on 16 January 2017.
« Link: Let's Stop Ascribing Meaning to Code Points
Making my machine stay responsive when writing to USB drives »

Page tools: View Source, View Normal, Add Comment.
Search:
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Mon Jan 16 01:32:16 2017
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.