When using Putty to remote into Linux machines this would happen to me ALL OF THE TIME. I'd be happily chugging along, editing some text in vi and my Windows instincts would kick in and I'd hit Ctr-S to save. Everything would freeze and I'd be sitting there wondering what in the hell had happened...
I never thought anything of it because in KDE terminals (konsole, yakuake) when you press CTRL+S, a yellow warning appears on the terminal stating that output is suspended and may be resumed by CTRL+Q. :)
This reminds me of a problem we had with running processes from python with subprocess.POpen. Some times after running the program for some time it just froze, none of us could figure out why and we reviewed the source of the invoked program and debugged it over and over again without finding anything that could get locked up. Eventually we found out that the problem was not in the invoked program, it was within python; piping the stdout into POpen was done to a fixed size buffer that was only being read/cleared after the invoked program exits, and since it can't continue to run it will never exit, stdout was getting blocked so just by writing to it the program froze. The solution to the problem was to start a new thread that kept reading the piped stdout and move it to another buffer. Today the documentation has a big warning about this behavior and a convenience-wrapper, communicate(), that does the threaded read for you but back then figuring this out was not exactly obvious.
Just to be clear, the issue is not in Python: the deadlock will happen if you won't consume the pipe whatever language you use to spawn a subprocess -- the buffer is not in your application -- it is the OS pipe buffer.
I imagine your broken code did something like:
#XXX WRONG, DON'T DO IT
p = Popen(cmd, stdout=PIPE)
p.wait() #XXX deadlock is possible if cmd generates enough output
output = p.stdout.read()
To avoid the deadlock, it is enough to swap the last two statements:
p = Popen(cmd, stdout=PIPE)
output = p.stdout.read()
p.wait()
that you would normally write as:
output = check_output(cmd) # 2.7+
If you need to consume more than one pipe separately then you should consume them concurrently and therefore you need threads or async.io.
`Popen.communicate()` can read from both pipes concurrently (as you've mentioned) and it is always available. It was available even in Python 2.4 that introduced the subprocess module.
Yep, I've hit this in Perl and Ruby and even once wrote a class to do something like Python's `communicate`. It's especially trappy if you want to run a command and both send stuff to its stdin and read stuff from its stdout. Bash makes it so easy, but it's a fraught with danger anywhere else!
Funny, I fixed a bug caused by exactly this recently in a program in production that had been humming along nicely until the output of another utility had grown larger than it ever was in testing or previously in production, causing the deadlock. The big warning in the Python docs sure was helpful in finding the problem.
Well that's interesting. There are two nearly identical keyboard mishaps that can wedge a screened process, and they both require ^Q to recover.
I've always attributed these hangs to the sad fact that hitting ^A can fairly easily come off as "^A ^S", especially when you do the keyboard hit with a slightly rolling hand motion. And as that key combination means "send ^S to the process inside screen", it results the inner process receiving SIGSTOP. Of course, the recovery is nearly the same as in the article: ^A ^Q.
For a screen user both have the same cargo-cult solution - just send enough SIGCONTs to the screen to get the application responding again.
I suspect that tmux chose ^B as the default command key to prevent accidental process hangs.
Either way, ^A is supposed to be "go to the start of the line" and it drove me absolutely nuts when I was using screen and had to ^A^A. There was probably a way to remap that, but I was introduced to tmux via byobu and now just use straight tmux on everything (works great on OSX too!)
^S and ^Q don't send signals; they make the tty driver stop delivering output. If you ^S a process that's not generating output, it will continue to run until it generates some output (until it fills an output buffer? not sure; the original intent, to match a fast process with a slow output device, would be well served by buffering some output).
SIGSTOP is like SIGTSTP, the signal usually generated by ^Z, except (like SIGKILL, signal number 9) it's not possible to catch it. It makes the process stop doing anything at all until something sends SIGCONT (which shows up in ps output as state T).
^A in bash will go to the beginning of the line. I don't know if this is why tmux changed the key, but it certainly makes a bigger difference to me because I use that function a lot. I rarely press ^S by accident (less often than I press it on purpose.)
I find it really cute that XON/XOFF are still with us, and people still marvel at the discovery from the deep dungeons of present time, since its pretty much everywhere in modern systems ..
Also of note, besides "TSTOP/TSTART" (^S/^Q) - as I knew it before XOFF/XON became the standard nomenclature courtesy of ASCII (X = Transmission), is of course the good ol' ^L, which is affectionately known as the boss-key in some parts. Life can't persist without some ^M and ^N business as well, and lets never forget the favourite "wake up your co-worker and go get coffee", ^G, although for some this may require your terminal have a buzzer or other suitable charge-based co-worker wake-up circuitry.
Well, I'm sure there are more in the alphabet to discover, but I'll just leave you here with the reminder that ^C and a few others of the happy little family of ^W control sequences are really but a few bytes away. ^Z
It really is still everywhere. For example, you can use ^S to pause your PC's POST display if you're fast enough. The Windows command line still uses it. And so on.
Conversely: I use ^S and ^Q _all the time_. I've got a long-running program that's spewing output to a terminal? ^S and it stops so I can look at it, copy output elsewhere, etc. ^Q and it resumes again seamlessly.
It occurs to me that I grew up on the BBC Micro, where pressing Ctrl+Shift would pausing scrolling, so being able to do this is tightly engrained in muscle memory...
Fair enough, though I tend to just use tee and/or tmux for things like that. Just turn logging on if I'm curious and let things continue on their merry way.
Only time i've ever used it was when booting a system to stop output on a serial console. That was years ago and a one off basically.
> Conversely: I use ^S and ^Q _all the time_. I've got a long-running program that's spewing output to a terminal? ^S and it stops so I can look at it, copy output elsewhere, etc. ^Q and it resumes again seamlessly.
Why not ^Z and `fg`? (I'm sure that there's a subtle or not-so-subtle difference, but I've only ever suspended, never XOFF'd, so I don't know it.)
^Z suspends the current application and returns to the program that ran it (typically the shell), which will then print a prompt. ^S and ^Q leave the output entirely undisturbed and unbroken.
Note that this enables you to use the bash 'forward search', CTRL+R's forgotten twin.
I use CTRL+R and CTRL+S often in emacs and was frustrated that CTRL+S would not work on a command line terminal. As the article mentions, CTRL+S is typically interpreted as XON. Turning that off will enable forward search.
I had a problem with scripts (reeeeeally hairy scripts with intertwined tcl and shell code, the tcl being used to drive some really obscure software) that failed when called from tcsh scripts which didn't have "-f" in them, and the user's .tcshrc had stty -ixon in it. I never figured out what the problem was...
I looked through his comment history and I really can't agree with you, this one comment is funny given the speed of todays consoles (when in the past I could easily keep up). Maybe you're confusing him/her with some other user?
I've only used flow control when booting systems like solaris on a serial console or other embedded hardware. Almost never for a tty session. I'm almost always saving output to a file and looking at that. Or using less +F or tmux etc... so yeah.
So my use of never might be hyperbolic, but it is a figure of speech. I can count on one hand with fingers leftover how often I've "needed" flow control. So just like in math, 0.99999 is equivalent functionally to 1 in this case.
I'm using an Arduino running GRBL for a project. It can take up to ~128 bytes of gcode in its buffer, and anything more gets dropped. So you have to carefully shepherd new statements in as they are executed (scanning for "ok".) Would be a perfect use for flow control, if arduino didn't already use it for something else. Dammit.
I'm not sure if you're already doing this, or if it's relevant to you, but a possible way to improve performance is to allow a "window" of commands that you haven't received an "ok" for. I believe common host software for RepRaps just allows one command at a time. How to define the window correctly depends on how the firmware does the buffering (in terms of bytes, or complete commands, or both).
This is absolutely one of those things I've had happen once or twice a year, poke around in puzzlement for 5 minutes, and then just restart and get on with my day. You knew it was a keystroke but never quite followed through on tracking down what/how to fix. Quite nice to know what the hell was going on all this time :)
I have been there... The problem is that this happens to just rarely enough I don't remember ^Q and randomly poke at the keyboard for a while until I hit the right combination.
Remember that holding control and any alphabet letter is supposed to have a function, and then remember "Quick" and "Stop".
Probably you've used Control-C a few times, maybe Control-Z too .. well, these basic characters have a control function over your process' terminal input/output, just like ^M/^N (carriage return/line-feed), ^L (clear screen) and so on .. Just like Control-C is a shortcut for "Cancel" process.. C-S-Q is sort of the 'gearing' over process output, where Z, which is 'detach process i/o from console', otherwise known as "background the process", ( * ) might be considered the 'clutch'.
Anyway, C-S-Q .. those 4 keys control the process. Cancel, Stop, Quick.
Practice it a few times with some cat /somelargefile.txt paging and you'll see there's a use for it beyond perplexity. When 'less' and even 'more' are not around, Control-S/-Q are your trusty pals.
(* - Another "oh, fudge, dunno what happened, guess I'll just reboot.." moment can occur with errant Control-Z usage over a process, where you accidentally put the process in the background and don't know how to re-attach it. In that case, the command 'fg' is your friend .. I've see far too many experienced people not know they can put their processes back in place during debug/admin sessions, opting indeed to reboot, or at least restart the term session, that I can't help but feel that, like Stop/Quick, Control-Z+'fg' is somewhat of a lost lore..)
I love ctrl-Z, but I think of it more as "suspend" than "background". After all, a ctrl-Z'd process isn't really doing anything. If you want to run such in the backround, just use bg, which is precisely analogous to fg.
Yes, you're probably right. It disconnects the console i/o from the process fd's, and so the process sits there blocking - and when you fg, the fd's get re-attached and things kick back into place .. while bg'ing the process, I think, re-attaches it to a new thread completely? Ah well, the details are out there somewhere ..
I hit the issue with blocking log IO all the time, not because of ^S, but because I develop in tmux and hit ^A[^B^B^B to read some stack trace (my ^A is tmux's ^B), then Alt-Tab to the browser, hit ^R, and it just spins. Then I wack my head again, Alt-Tab to tmux, hit Enter, and the page loads.
When coming from windows world, I quickly learned that ctrl+s can be undone with ctrl+q. And that's all, never went into more detail. Wrote it off as a quirky stuff linux has plenty of.
Personally I use Ctrl-S for tmux for the command key, since I use Ctrl-A in vim. If I really need to send a signal I can Ctrl-s Ctrl-s, but otherwise this works fine.
I always seem like a genius when I ctrl-q out of someone's inexplicably locked terminal. Lord knows I discovered it through some serious pain and wonderment.
Apparently this is a fairly common problem. Take a look at: https://raam.org/2007/recovering-from-ctrls-in-putty/ The scrollbar should give some indication of how many "Thank You's" this article warranted!