A fun Linux kernel pty problem, or maybe it's not a problem

March 6, 2011

Every so often I run into a Linux kernel 'bug' where I am not sure if it actually is a bug, or at least where I am not sure if the Linux kernel people would agree with me about it being a bug. Recently, one came up with the pty handling code in recent kernel versions (I don't know how far back it goes, but it's present in 2.6.32 in at least some configurations).

I have a sort of terminal emulator program that wants to allow you to edit the text of the line you're entering 'locally', inside the terminal emulator, instead of having to rely on your shell or program to provide line editing. As part of this, it needs to write your final line of input to the pty without having it echoed (because it has already made it visible to you as you were writing it); however, it wants to leave the underlying pty terminal mode looking as normal as possible for as long as possible, in case a program that you're running cares.

If the program didn't care about keeping pty modes normal, it could just turn off ECHO on the pty. Since it does care, it leaves ECHO on normally; when you tell it you're done and the input line should go to your shell or whatever else is running on the pty, it turns echo off on the slave side of the pty, writes the input line to the master side of the pty, and then turns echo back on on the slave side of the pty.

On old versions of the Linux kernel and on other Unixes, this works and it acts as if the sequence of operations are packetized (across both the master and slave sides of the pty); there is a 'turn echo off' packet, a 'here is some data' packet, and a 'turn echo on' packet, and they are all processed in order. Since the data packet is completely processed before the 'turn echo on', nothing from it is echoed. On current versions of the Linux kernel, some or all of the input line is echoed; how much is echoed depends on timing. The kernel is now acting as if 'turn echo off' and 'turn echo on' are instant operations that take effect the moment you make the system call; they are not interleaved with data operations. If there is unprocessed input data when echo is turned on, it gets echoed.

(You can find a test program for this here.)

On the one hand, this is a behavior change from prior Linux versions and from what other Unix versions do. On the other hand, it sort of feels as if this has always worked by accident since the more I look at it, the more peculiar it feels. So the net result is that I am not sure that the new behavior is actually a bug, instead of just a change that stops doing something odd and wacky that previously worked more or less by accident.

Written on 06 March 2011.
« FastCGI's encoding mistakes and how not to design a wire protocol
What terminal emulators I use when »

Page tools: View Source, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Sun Mar 6 02:25:09 2011
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.