This is pretty huge. For as long as I can remember the response to command line applications talking to command line applications was "Why would you want to do that? Use (RPC | shared memory | some other IPC mechanism)." And nobody at Microsoft seemed to understand how much simpler it was to use ptys. They seem to have completely capitulated to the notion ptys and are dropping them into the next release of W10. I wish this had happened 10 years ago but hey, I'll take it.
It's the reality of the market, which is why Windows is adding Linux compatibility (as is every *BSD, Illumos, ...).
But also it's the fact that three decades of not even life support has left the Windows console in pretty sad shape -- the folks tasked with getting it into better shape were bound to see the value of ptys.
Lastly, don't forget that Windows NT was meant to be a console OS, like VMS. There must still be people, even if very few, at MSFT who appreciate text-oriented apps.
For me, the tty/pty, shells, screen/tmux/..., ssh, and so on, are the things that make Unix so powerful. The fact is that Win32 is far superior in a number of areas (SIDs >> UIDs/GIDs, security descriptors >> {owner, group, mode, [ACL]}, access tokens >> struct cred), but far inferior in the things that really matter to a power user trying to get things done.
> Lastly, don't forget that Windows NT was meant to be a console OS, like VMS. There must still be people, even if very few, at MSFT who appreciate text-oriented apps.
I expect that, like Linux compatibility, most of it is not about "apps" but about being better at running in the cloud, where a (virtual) machine or container needs to be as light as possible, and to be configured and a service launched in it as unattended/automated manner as possible. Stripping out the GUI and making command lines work better works towards these goals.
I understand this as a "compatibility with * nix" existing software, not as an "amazing feature." Can anybody suggest why I should like it, except for the compatibility with the software written for * nix terminals? Even ssh being too interconnected with a terminal down to the many details was a shock for me... I expected the simple encryption mechanism, over which whatever communicates, even if that would implement the "sh" part of it only on top of that, but no, it's everything spaghettisized and recombined with everything unnecessary like emulating the devices that don't exist for many decades -- in something that should have a clear separation between the task of transporting encrypted data with authenticating from anything else. I can't see it as being positive, security-wise.
Isn't it strange that today everybody has very powerful GPUs and CPUs and the graphical displays with immense RAM and then using all that to emulate the terminals last existing decades ago appears to be so important, even for something that should be just a secure communication protocol?
Why do we still spend so much energy to decide which console of many decades ago we "support" when it seems that all are flawed, at least compared to what the modern OSes can provide, as soon as the "compatibility" is not needed?
Isn't all that "hardware console" compatibility stuff just a historical accident from the "bad old days" of 300 baud lines between the mainframe and the "terminal" which had a few bytes of RAM total? In the days when e.g. the Thunderbolt 3 can carry 5 GB/s, and the rest of the hardware matches? Why do people still so cling to it? I'd really like to know what I am missing.
> I expected the simple encryption mechanism, over which whatever communicates
In the UNIX world, that's what it gives you - a stream of bytes. Hence things like rsync-over-ssh or git-over-ssh. It also has a port forwarding mode which has special support for X11, which gives you remote windowing over a stream of bytes too.
The main, huge, benefit is that the abstraction is pretty simple, it's discoverable, and you can use the same interface as a human. You can also plug any stream-of-bytes into any other stream-of-bytes, whereas API or RPC based systems have to be designed to interoperate.
As I’ve tried to implement my minimal ssh client (just to connect, execute some command and get the result) I’ve had exactly opposite impression of the “just a stream of bytes” that you mention -- exactly the lack the abstraction. Can you point to any source that does ssh without having to care about a lot of weird terminal and console ancient stuff? I’d be really glad to see it! To me it looked as “everything and the kitchen sink” (that is, exactly the kind of things mentioned in the OP or the comments, like terminal signals and whatnot) has to be there.
SSL is straightforward compared that, at least, once the keys are set. But ssh... as seen in the OP even the console or the terminal or however that part it called has to be very special, and they are obviously proud they implemented that too. In 2018. Probably decades after the last single hardware terminal was sold.
I see your point - but don't you find it kinda charming? It's a bit like drinking tea in a china cup, or eating a scone with clotted cream.
I think there is probably a lot of room for improvement in the terminal world, and I agree that a lot of the really old stuff makes things a bit counter-intuitive, but for whatever reason, it seems that people who make really good software also tend to be the people who are pretty fanatical about backwards compatibility. Consider vim, for instance.
In the end, I think the basic model of interoperable, small programs that manipulate streams of text is really good - so people will put up with any number of weird rituals to live in that model. It's also very humble, and very unexciting, so it's the kind of thing that's hard to get people hyped about. So it probably filters by the people that like old things.
You are conflating what SSH does with what the terminal you're running it in does. SSH barely knows what terminal emulation is. You can literally pipe binary data into it at one end and receive that data into a program at the other end. That's what programs like scp and rsync do. They launch ssh and use it as a data channel.
That is from a different family of operating systems to what is headlined here, and does not have the client-server system of the operating system family discussed.
Will there be a terminfo database entry for ConPty? What TERM string should we expect to see?
To elaborate: although an ordinary POSIX pty doesn't inherently have a terminal type - that's entirely down to whatever emulator is connected to the master side - the way the ConPty system translates Console API calls into terminal control codes means that it necessarily needs to pick a terminal emulation, which all actors in the ConPty system are expected to use.
A terminfo database entry would be useful both for applications running on *NIX hosts but displaying on a remote ConPty master somewhere, as well as for porting existing terminal applications to Windows where they will run on a ConPty slave.
As a follow-up question, presumably this means that the SSHD running on Windows as a ConPty master needs to translate between whatever terminal emulation the ssh client is connected to and the one expected by ConPty / ConPty apps (in the same way it must translate between the native ConPty UTF-8 and the remote charset)?
Commandline applications on linux rely on a TERM setting (with termcaps) to be able to know what VT sequences the terminal is able to support. On Windows, we only really have one terminal, conhost.exe, and our goal there is to be compatible with TERM=`xterm-256color`. That's why you'll see that WSL has that set as the default term setting.
Now even with ConPTY, when a client writes VT sequences, they still need to be interpreted by conhost. This is because technically, a console application could use both VT and the console API, and we need to make sure the buffer is consistent. So clients should still assume that they should write out `xterm-256color` compatible sequences.
Now on the other side of thngs, the "master"/terminal side of conpty, we're going to "render" the buffer changes to VT. Fortunately, we'd dont really need a deep VT vocabulary to make this possible, so the VT that's coming out of a conpty is actually pretty straightforward, probably even vt100 level (or I guess vt100-256colors, as insane a termcap that would be).
It's definitely a future feature that we'd like to add to make conpty support multiple different TERM settings, and change the sequences we emit based on what the terminal on the other side is going to expect.
We haven't really gotten into the nitty gritty of all of this quite yet, so if you find bugs or have feature requests, we're happy to take a look at them. You can file issues on [our github](https://github.com/microsoft/console) and we'll add them to our backlog
Thanks. It makes sense to pick an existing terminal like xterm-256color to target - that way you don't have to worry about a new terminfo database entry getting distributed out.
The nitty-gritty can get quite nitty - things like bracketed paste and set window title.
That would be sad, as there is already far too much software out there that hard-codes escape sequences, completely ignoring the TERM environment variable.
Even worse, sometimes they won't even disable escape codes when they should not be displayed.
I've posted bug reports for very popular software packages whose commandline always output vt102, even when TERM is set to dumb or when run through pipes. That makes grepping for error messages somewhat annoying. In at least some cases these reports were ignored.
What next? Job control signals?? :) (EDIT: How about tmux?)
Anyways, this is fantastic. Finally, proper ssh functionality!
This will encourage development of console (text-oriented) apps for Windows, which I hope will be much simpler. Interfacing with the console can be really difficult if you're coming from *nix. Ideally all the WIN32-specific code in, e.g., jq[0], could be ripped out.
Windows does have signals! It just splits them into a few facilities. POSIX "synchronous" signals correspond to SEH exceptions and can be handled roughly the same way --- except that signals have process global handlers and Windows has thread-local ones, because the glibc people are sticks in the mud and are hostile to any attempt to make signals suck less.
For asynchronous signals, like SIGINT, Windows create a new thread out of thin air to deliver your app a notification. That's not really all that much better than a signal from a concurrency perspective.
Windows even has APCs, which are like regular signals that are delivered only at explicit system call boundaries.
Every operating system needs some mechanism to tell a process to do something. Windows has evolved an approach that isn't all that different from Unix signal handling.
Spawning a new thread to handle a signal is much better than preemption: you then have no concerns about async-signal-safety that aren't plain old thread-safety concerns. I'd much rather have thread-safety constraints than async-signal-safety constraints.
Processes. A quick skim of the documentation linked by the person I answered suggests that processes can be associated with a job object, but threads cannot. But a process cannot really be suspended directly, except by suspending all its threads.
First, tmux works fine in WSL. Secondly, if you like both tmux and Windows, there's a fair chance you'll like ConEmu's split panel facility even better. It's basically tmux, but more "windowsy". ConEmu is spectacular.
For me, the most surprising thing is that the new PTY devices use UTF-8. Not UTF-16 or UCS-2 or weird little endian variants thereof, and not even wchar_t.
It is! But this is a very un-windows like feature, isn't it? We want this to work on other platforms with as little modification as necessary, and frankly, jumping through the wchar_t<->char hoops is a _pain_. So we'll do it for you!
Hear hear! wchar_t is a disaster. UTF-16 is terrible. I'm not at all convinced that 2^21 codepoints will be enough, so someday it'd be nice to be able to get past UTF-16 and move to UTF-8, and Windows and ECMAScript are the biggest impediments to that. Your choice of UTF-8 will tend to place UTF-8 on a level playing field in Win32.
I guess, too, that this is the end of codepages -- I doubt they'd go away, but there should be no more need to struggle with them, just use UTF-8. You'll still need a semblance of locale, for localization purposes, naturally, but all-UTF-8-all-the-time is a great simplification.
Confused, where does 2^21 code points come from and how is that related to the UTF-16 vs. UTF-8 distinction? Can't both of them encode all Unicode code points? Or are you thinking of code units perhaps, and UCS-2? Although even there I'm confused where the 2^21 came from.
UTF-16 has a limit on the size of a code point because a code point is either a single normal code unit or a pair of surrogate code units, each encoding 10 bits of the code point (I think I used the right terminology). UTF-8 has a natural extension path to up to 7-byte encodings with all the usual UTF-8 properties (first code unit indicates how many remain, other code units are recognizable as not the first).
Where are you getting this information though? I haven't worked out the bits myself yet but Wikipedia's first sentence itself says UTF-16 can encode all 1,112,064 valid code points of Unicode, which is already more than 2^(10+10) = 1,048,576.
Unicode code point space: Was 16-bit (0000 to FFFF), then 32-bit (00000000 to FFFFFFFF), and is now 21-bit (00000000 to 0010FFFF)
UTF-16: Encodes the entire 21-bit range, encoding most of the first 0000 to FFFF range as-is, and using surrogate pairs in that range to encode 00010000 to 0010FFFF. The latter range is shifted to 00000000 to 000FFFFF before encoding, which can be encoded in the 20 bits that surrogate pairs provide. This is a subtlety that one likely does not appreciate if one learns UTF-8 first and expects UTF-16 to be like it.
UTF-8: Could originally encode 00000000 to 7FFFFFFF, but since the limitation to just the first 17 planes a lot of UTF-8 codecs in the real world actually no longer contain the code for handling the longer sequences. Witness things like the UTF-8 codec in MySQL, whose 32-bit support conditional compilation switch is mentioned at https://news.ycombinator.com/item?id=17311048 .
I don't see anything wrong with what you're saying, but I still don't get how it explains the original comment I replied to [1]:
> I'm not at all convinced that 2^21 codepoints will be enough, so someday it'd be nice to be able to get past UTF-16 and move to UTF-8
UTF-16 currently uses up to 2 16-bit code units per code point, whereas UTF-8 uses up to 4 8-bit code units per code point, and the latter wastes more bits for continuation than the former. How is "getting past UTF-16 and moving to UTF-8" supposed to increase the number of code points we can represent, as claimed above? If anything, UTF-16 wastes fewer bits in the current maximum number of code units, so it should have more room for expansion without increasing the number of code units.
If I had intended to explain that, I wouldn't be replying to your comment about your not working out the bits with an explanation that works out the bits for you, and shows that it really is capable of encoding all 17 planes even though surrogate pairs have only 20 bits.
And as you can see, if you do work out the bits, you find that cryptonector is wrong, since UTF-8 (as it has been standardized from almost the start of the 21st century, and as codecs in the real world have taken to implementing it since) encodes no more bits than UTF-16 does. It's 21 bits for both.
No, UTF-8 could, and used to, encode much more than 21 bits of codepoint space. It has been artificially limited to better match UTF-16 -- UTF-16's limits are not artificial but fundamental. If some day we need more bits, we'll simply obsolete UTF-16 and drop those limits on UTF-8. MSFT seems to be taking steps to put UTF-8 on a level or even higher playing field than UTF-16. We should welcome this.
It seemed like you were trying to correct my comment, but everything you said seemed to support what I was saying, so I thought maybe you were trying to continue the initial discussion... I guess not.
With regards to the comment then: the range downshifting you mentioned is merely a step in the encoding process though -- the code point is still whatever it was. If you read parent comment, it had claimed that, in a surrogate pair, each of the 2 code units encodes 10 bits of the code point... but that would be missing 1 bit when the code points need 21 bits to be represented. That's all I was saying there. The extra bit indicating that it's in fact a surrogate pair isn't some kind of implicit dummy bit that you can pretend isn't encoding anything -- if it wasn't there then clearly it wouldn't be encoding the code point for a surrogate pair anymore.
Yes, UTF-16 can encode all currently valid Unicode codepoints, which is more than 2²⁰ but less than 2²¹. But cryptonector doesn't believe it will be enough in the future.
OTOH, UTF-8, as originally defined, can encode 2³¹ codepoints.
UTF-16 is not always 16b/codepoint. UCS-2, its ancestor and more or less what is actually used by Windows when it talks about "Unicode", was 16b/codepoint. It turned out that 64k codepoints wasn't enough for everybody, so they increased the range and added "surrogate pairs" to UCS-2 to make UTF-16. Codepoints in UTF-16 are, therefore, either 16b/codepoint or 32b/codepoint. Many systems that adopted UCS-2 claim UTF-16 compatability but in fact allow unpaired surrogate pairs (an error in UTF-16). The necessity of encoding such invalid
UTF-16 strings in UTF-8 under certain circumstances lead to the pseudo-standard known as WTF-8[0].
UTF-8 was designed (as legend has it, on the back of a napkin during a dinner break) after the increase in range and doesn't suffer from the same problem. Additionally, it's straight-forward "continuation" mechanism isn't any more difficult to deal with than surrogate pairs, and it doesn't have any endianess issues like UTF-16/UCS-2.
UTF-16 is garbage. Windows is stuck with it because it was too early an adopter of Unicode. Oh the irony. This may set Windows on a path to deprecating UTF-16 -- godspeed!
Another exciting development in moving beyond UTF-16: Microsoft is experimenting with adding a native UTF-8 string in .NET next to the existing UTF-16 string:
I've just spent some time reading through the proposals, it made for a fascinating read! It's really interesting to see the work and discussions that go into a seemingly simple feature like this.
Could you elaborate? I've been under the guise for most of my career that doubling a digit leads to huge benefits that I'm too comp-sci ignorant to understand.
Can I assume that's just subtle humor on your part?
If not:
UTF-16 is born of UCS-2 being a very poor codeset, as it was limited to the Unicode BMP, which means 2^16 codepoints, but Unicode has many more codepoints, so users couldn't have the pile-of-poo emoticon. Something had to be done, and that something was to create a variable-length (in terms of 16-bit code units) encoding using a few then-unassigned codepoints in the BMP. The result yields only a sad, pathetic, measly 2^21 codepoints, and that's just not that much. Moreover, while many codesets play well with ASCII, UTF-16 doesn't. Also, decomposed forms of Unicode glyphs necessarily involve multiple codepoints, thus multiple code units... Many programmers hate variable length text encoding because they can't do simple array indexing operations to find the nth character in a string, but with UTF-8, UTF-16, and just plain decomposition, that's a fact of life anyways. If you're going to have a variable-length codeset encoding, you might as well use UTF-8 and get all its plays-nice-with-ASCII benefits. For Latin-mostly text UTF-8 also is more efficient than UTF-16, so there is a slight benefit there.
Much of the rest of the non-Windows, non-ECMAScript world has settled on UTF-8, and that's a very very good thing.
No humor whatsoever. Thank you for the explanation! I'm an ops person who knows python/golang to a dangerous extent and have never gone out of my way to understand the UTF reasonings. Your post intrigued me and made me want to ask why you felt that way. This will make me sound horrendously ignorant to someone of the likes of someone such as yourself but I'm here to learn.
UTF-16 has a bit of a funky design (using four byte/two code unit surrogate pairs to encode code points outside the basic multilingual plane) that ultimately restricts Unicode (if compatibility is to be maintained with UTF-16, at least) to 17 planes, or 2^20 code points (about 1 million).
UTF-8 uses a variable length encoding that allows for more characters-- if restricted to four bytes, it allows for 2^21 total code points; it's designed to eventually allow for 2^31 code points, which works out to about 2 billion code points that can be expressed.
(Granted, this is all hypothetical-- Unicode isn't even close to filling all of the space that UTF-16 allows; there aren't enough known writing systems yet to be encoded to fill all of the remaining Unicode planes (3-13 of 17 are all still unassigned). But UTF-16's still nonstandard (most of the world's standardized on UTF-8) and kind of ugly, so the sooner it goes away, the better.)
That is a bit misleading to the point of error, on several points:
* Your timeline is backwards. UTF-8 was designed for a 31-bit code space. Far from that being its future, that is its past. In the 21st century it was explicitly reduced from 31-bit capable to 21 bits.
* UTF-16 is just as standard as UTF-8 is, it being standardized by the same people in the same places.
* 17 planes is 21 bits; it is 16 planes that is 20 bits.
Joel Spolsky's essay "The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!)" is an excellent read:
Wikipedia says: "UTF-8 requires 8, 16, 24 or 32 bits (one to four octets) to encode a Unicode character, UTF-16 requires either 16 or 32 bits to encode a character"
> I've been under the guise for most of my career that doubling a digit leads to huge benefits that I'm too comp-sci ignorant to understand.
I was confused about this for years, too. But it turns out it's just a problem of bad naming. Happens more in this industry than we'd like to admit.
As other explained, it boils down to UTF-16 being 16-bit, and UTF-8 being anything from 8- to 32-bit. It should have been named UTF-V (from "variable") or something, but here we are.
UTF-16 is a variable-length encoding using up to two code units which each are 16-bits wide.
UTF-8 is a variable-length encoding using up to 4 code units (though it used to be up to 6, and could again be up to 6) each of which are 8-bits wide.
Both, UTF-16 and UTF-8 are variable-length encodings!
UTF-32 is not variable-length, but even so, the way Unicode works a character like ´ (á) can be written in two different ways, one of which requires one codepoint and one of which requires two (regardless of encoding), while ṻ (LATIN SMALL LETTER U WITH MACRON AND DIAERESIS) can be written in up to five different ways requiring from one to three different codepoints (regardless of encoding).
Not every character has a one-codepoint representation in Unicode, or at least not every character has a canonically-pre-composed one-codepoint representation in Unicode.
Therefore, many characters in Unicode can be expected to be written in multiple codepoints regardless of encoding. Therefore all programmers dealing with text need to be prepared for being unable to do an O(1) array index operation to get at the nth character of a string.
(In UTF-32 you can do an O(1) array index operation to get to the nth codepoint, not character, but one is usually only ever interested in getting the nth character.)
For completeness sake: the confusing part is that there is (used to be) a constant-length encoding that uses the exact same codepoints as UTF-16, but doesn't allow the variable-length extensions. That encoding is called UCS-2 and although deprecated, is the reason why many people think UTF-16 is constant-length.
I suppose it's mostly TUI programs, which use low level console API. So did you try to capture output of something like Far Manager[0]? If so, will it be much simpler to parse escape sequences of VT100?
Hey I'm one of the Console devs who's been working on this feature for a while now. I'll be hanging around in the comments for a little while to try and answer any questions that people might have.
TL;DR of this announcement: We've added a new pseudoconsole feature to the Windows Console that will the people create "Terminal" applications on Windows very similarly to how they work on *nix. Terminals will be able to interact with the conpty using only a stream of characters, while commandline applications will be able to keep using the entire console API surface as they always have.
While it is nice that MS is focusing on console and command line now, it seems to me that you are mostly working on improving compatibility with legacy UNIXy stuff.
Do you have some vision or plans to go well beyond the classic UNIXy style of console and command line? I'm thinking in the lines of projects like DomTerm http://domterm.org/ which could have nice interactions with e.g. PowerShell.
I haven't seen DomTerm before, but it looks pretty awesome. At a glance, it's basically a GUI-fied tmux hosted in Electron? It would be awesome to have in Windows, but wouldn't that just require that DomTerm add support for these ConPty APIs?
In any case, I'm more interested in your proposed interactions. Did you have anything cool in mind? Given that we ship PowerShell on Linux, we could theoretically do some stuff there (including within PowerShell on WSL) before it's hooked up to ConPty
I'm not the person you were asking, but this should interest you.
I've been working on a terminal emulator ( Extraterm http://extraterm.org/ ) with some novel features which would dovetail nicely with how PowerShell works. The first is the ability to send files to the terminal where they can be displayed as text, images, sound, etc or as an opaque download. Extraterm also adds a command `from` which lets you use previous terminal output or files, as input in a command pipeline. See http://extraterm.org/features.html "Reusing Command Output" for a demo. This opens up other, more interactive and iterative workflows. For example, you could show some initial data and then in later commands filter and refine it while visually checking the intermediate results.
What I would like to do sometime is integrate this idea with PowerShell and its approach of processing objects instead of "streams of bytes". It should then be possible to display a PowerShell list of objects directly in the terminal, and then reuse that list in a different command while preserving the "objectness" of the data. For example, you could show a list of user objects in one tab and then in another tab (possibly a different machine) grab that list and filter it the same way as any normal list of objects in PowerShell. You could also directly show tabular data in the terminal, let the user edit it "in place" in the terminal, and then use that editted data in a new command. It allows for more hybrid and interactive workflows in the terminal while still remaining centered around the command line.
Extraterm does these features using extra (custom) vt escape codes. ConPty should allow me to extend these features to Windows too.
Ooooh yeah, that sounds awesome! Going to share this with some people on our team (lots of folks love and use Hyper already, but we're always looking for new stuff to play with).
I would highly recommend you check out the excellent HistoryPx module[1]. Among (many) other things, it supports automatically saving the most recently emitted output to a magic `$__` variable. Theoretically, you could save a lot further back, but you may start to run into memory constraints (turns out .NET objects are a little heavier than text... ;) )
The ConPTY is not just about compatibility with nix. It's about proper remoting of consoles. Unix got that right / Windows got that wrong, and now Windows will finally get it right too -- that it helps nix compat seems like a happy accident (though obviously they want that too, so not so accidental.
Right. For a long time, the MS remoting philosophy was that applications should be remoted, not text streams. The stance goes all the way back to DCOM. That's why PowerShell remoting looks more like a local PowerShell executing commands on a remote machine than it looks like you just connecting to a PowerShell running elsewhere.
The difference is important, since in the traditional MS model, each program that wants to do the remote thing needs to essentially implement its own client-server setup, albeit with a massive amount of help from various runtimes. Named pipes and central authentication made this approach not quite as horrible as it sounds.
This new API is a departure from this model. It will make it possible to just remote via text streams. Perhaps that's uglier --- everyone knows in-band signaling is fragile. But long experience shows they just remoting the damn text streams is easily the more pragmatic option.
I mean, it could be a packet stream where some packets are control packets, and then the Console API could be used and remoted, but you'd need clients that understand it.
The fragility of in-band signaling in the TTY world is not *nix's fault or anything. VMS had that too, since VMS too had to deal with TTYs. The fact is that a) the system evolved from real, hardware TTYs of the 1970s, b) using a text stream with some in-band signaling doesn't even half suck -- mostly it rocks, and all you have to do for it to rock is get the right $TERM value and not output binary files to the tty.
The Windows Console was even more terminally dead. Terminals have their limitations, but honestly, I'll take a terminal, tty/pty, and terminfo, any day over the Windows Console.
And now you're reinventing the Windows Console API... :)
I mean, both approaches have their pluses, but the API approach is only ever going to work well for remoting if it is standardized and interoperable. And the installed base of Unix termcap/terminfo programs is huge, so plain old text-with-in-band-controls is not going away anytime soon.
The Windows way would be to mount the remote filesystem and use local mv and cp. Which works fine, since Windows has excellent remote filesystem support. The Unix world only partially began to catch up to CIFS with the advent of NFSv4 and its lease infrastructure and it still isn't as seamless as the Windows stuff.
Have you given thought on how to solve the "unwanted console" problem? For example if you run a .py file (Python) under Windows then you get a console. That is fine for command line stuff, but beyond annoying if the file displays as a gui. So there are now two Python binaries - python.exe and pythonw.exe. The only difference is the latter ensures no console appears. Also good luck if the script printed a help message since often the console disappears before you even know that happened.
I presume many tools deal with this issue, and do it in different ways. Perhaps it is as simple as making the console itself only appear once there is any output, or a blocking read of input.
Technically, any executable that's compiled as a commandline application is going to get a console allocated for it, no matter what on Windows. I don't believe that's something we can fix retroactively unfortunately, that's just a part of how things have to be.
Now, I believe that python could have python.exe compiled as a win32 application, then call AllocateConsole as soon as the script called print() or something. If the app was already running in a console, I believe (don't quote me) that AllocateConsole won't allocate a new console for it, but if it doesn't yet have a console it'll spawn one.
I think the behaviour of cmd.exe is part of the problem here. When an interactive cmd.exe launches a console-subsystem app, it waits for the process to finish before showing the prompt again, but when it launches a GUI-subsystem app, cmd.exe writes the prompt again immediately, so even if the new process calls AttachConsole(ATTACH_PARENT_PROCESS) before it tries to write to the console, it will write over cmd.exe's prompt, which makes a poor user experience.
So, if someone wants to make a "dual-mode" app that works as a win32-subsystem app when launched from Explorer and a console-subsystem app when launched from a console, they have to choose between two bad options. They can make their app a console-subsystem app, which means a console will always briefly appear on screen when the app is started (no matter how quickly the app calls FreeConsole(),) or they can make their app a GUI-subsystem app (that opportunistically calls AttachConsole(),) which behaves sub-optimally in cmd.exe.
Maybe the solution is to add a flag (in the .manifest file?) that makes the console initially hidden for a console-subsystem app. That would prevent the brief appearance of a console window when launching a console-subsystem app from Explorer. Then there would be no need for pythonw.exe and python.exe could show the console window only after a message is printed.
If AllocConsole does behave in the way you say (which I understand it may{, not}), then the documentation sorely needs updating, because right now that bit of functionality (if it is there) is rather implicit.
It would be really cool to effectively deprecate the current console functionality and make it relatively straightforward to use the PTY API going forward, adding the bits people need to support use cases like this (allocating a console when/as it's needed).
Perhaps Visual Studio could introduce a new template for commandline applications that targets the PTY, and put "(Recommended)" next to that one? :D
> It would be really cool to effectively deprecate the current console functionality and make it relatively straightforward to use the PTY API going forward,
Please no!
The console subsystem in Windows is what terminal I/O evolved into, in the 1980s. Going back to terminal I/O is a massively retrograde step.
In any case, for addressing the problem at hand removing the console API is not the answer. rossy has explained what the problems actually are, which relate to the command interpeter waiting for processes to terminate and whether Win32 program image files are marked as GUI or console.
Personally I have always regarded that behaviour of Microsoft's command interpreter as a flaw, not a feature. I've always turned it off in JP Software command interpreters, which make it configurable. I didn't implement it in my command interpreter. However, I do appreciate that Microsoft's strong commitment to backwards compatibility hampers what can be done.
Hmmm. Good point. I can't say I think the PTY architecture is actually fun or anything, it's brain-numbing. Now I realize what I was asking I disagree too, hah!
IMO, the "real" answer - the viable one :) - is a redesigned console model that supersets/encapsulates what already exists in such a way that things remain backward compatible but incorporate features that allow for progressive enhancement. The question is in how to actually build that out; what to start with, what to do when and where, etc.
Probably the most important thing I'd start with is having each console be like an independent terminal server: make it so anything can watch all the stdio streams of everything attached to the PTY, make it possible to introspect the resulting terminal stream, etc. Then the TTY itself could be queried to get the character cell grid, query individual characters, etc. And also make it possible to change arbitrary PTY+TTY settings [out from under whatever's using a given console] as well.
By "anything can watch" I mean that there would be an actual "terminal server" somewhere, likely in a process that owned a bunch of ptys, and this would have an IPC API to do monitoring and so forth. Obviously security and permissions would need to be factored in.
But this would roughly take the best from the UNIX side (line disciplines are kind of cool, having the PTY architecture Just Work with RS232 is... I understand the history, but it makes for an interesting current-day status quo, IMO), and then combine this with the best of the Win32 side (reading what's on the screen!!!!! Yes please!!).
I'm not sure how to build something sane that could incorporate graphics though. ReGIS and Sixel are... no. 8-bit cleanness is an unfortunately-probable requirement for portability (at least UTF-8 can be shooed away in broken environments with a LC_ALL=C), but base64 encoding is also equally no. Referencing files (what Enlightenment's Terminology does) is a nononono. w3m-image's approach of taking over the X11 drawable associated with the terminal window is awesome in its hilarious terribleness. The best I can think of is a library that all image/UI operations would be delegated to, which would do some escape-sequence dances with the terminal (and whatever proxies were in the way?) to detect capabilities and either use out-of-channel communications (???) to send the image data over rapidly, or alternatively 8-bit or 7-bit encode the image data into the TTY stream (worst case scenario).
This is a bit hazy/sketchily laid out, but it's something I've been thinking about for several years. When I started out pondering all of this stuff circa 2006 I was most definitely all over the place :) I'm a bit better now but I still have a lot of unresolved ideas/things. I'm trying to build a from-scratch UX that provides a more flexible model to using terminals and browsing the web, but in a way that's backward-compatible and not "different to the point of being boring".
I've (very slowly...) come to understand that slow and progressive enhancement is the only viable path forward (that people will adopt), so I'm trying to understand the best way to do that.
Actually, extending it in a native Win32 way in line with the existing console model would have been quite a simple thing.
There are already CONIN and CONOUT objects. One simply needs to make the latter a synchronization object, the former already being waitable. The console screen buffer maintains a dirty rectangle of cells that have been dirtied by any output operation, be it high-level or low-level output. It becomes signalled when that dirty rectangle is not zero-sized, the cursor is moved, or the screen buffer is resized. And there's a new GetConsoleScreenBufferDirtyRect() call that atomically retrieves the current rectangle and resets it to zero.
With that, capturing console I/O is simply a matter of waiting for the console output buffer handle along with all of the other handles that one is waiting for, getting size/cursor info and clearing the dirty rectangle with GetConsoleCursorInfo()/GetConsoleScreenBufferInfo()/GetConsoleScreenBufferDirtyRect(), and reading the new cell values of the dirty rectangle with ReadConsoleOutput().
Why... can't you have I/O redirection to the null device and have it understand (and throw away) Console API messages? You might as well also add some pseudo-device to convert Console API messages into Unix-style text streams (with or without metadata converted to terminal control sequences), so that one could redirect console programs' output to files / pipes.
When the user's (programmer's) intent is to run a program with no console window, then that's what they should get: no console window.
If I remember correctly, this approach breaks if stdin/out is redirected.
There's also some funky stuff about explicit AllocConsole-allocated consoles; for example, when you attach a native debugger, all output from such console is automatically redirected to that debugger (i.e. the VS Output window or similar). This is very annoying in practice.
I'm unclear as to whether I'll be able to pipe binary data between a classic console application and a ConPTY application due to the VT translation and rendering components in ConHost.
So, for example if I was to pipe into 7z.exe, a classic console app, using something like "type mybinaryfile.bin | 7z.exe a -si c:\temp\myarchive.7z" from a ConPTY console, would the VT translation affect the piped stream?
Nope! We're only rendering the effect of any attached processes to the VT on the conpty side of things. On the client side (where cmd, type, 7z.exe are all running), they're going to keep working just the same as they always have. They're all running on the "slave" side of conpty, while the emitted VT is coming from the "master" side of the conpty.
Those sound like they'll be more like the responsibility of the terminal emulator, unfortunately.
Windows console applications aren't really able to live without being attached to a console. Now, a terminal might be able to implement those features...
actually now you've got me thinking. I'll play around with that idea. Definitely non-committal, but it might be possible in the future.
The sad thing is that this was all already implemented and done in Microsoft's second POSIX subsystem for Windows NT. It provided signals and process groups support for job control shells. It had a full control sequence interpreter for output and control sequence generator for extended keys. There were termcap/terminfo database records that people had added to other operating systems. It had a line discipline with "canonical" and "raw" modes. It had pseudo-terminals, with both BSD and System 5 access semantics.
I wish people would stop pointing at the Linux subsystem and using it as an example of the stuff Windows in general could do. The Linux subsystem is a subsystem. In NT terminology, that makes it practically a container. It's disconnected from the rest of the system. It can interact with the win32 world in only a few constrained ways. You can't deliver SIGWINCH to a win32 process!
I mean, the same is true on Unix. If an app is in the background and wants to write to (or read from) the terminal, it gets SIGTTOU (SIGTTIN) sent to it immediately. It might have to be impossible on Windows to ignore SIGTTOU/SIGTTIN/SIGTSTOP, but I think that's just fine.
Mind you, ptys + tmux/similar is certainly very good, and if that's all we'll get that's still way way better than the current state of affairs, but if that's all that will be possible it should at least be possible to pause the console's output (and flow-control the console application).
Excellent question! There are a few limitations that we have to place on the ConPty to make it work quite right. Primarily, client apps running attached to the conpty will not be able to have separate viewport and buffer sizes. On *nix, the entire "console buffer" is just the size of the window, but on Windows, technically, the buffer is much larger than the window. (as an example, when you open up a command prompt, there's a giant empty space at the bottom if you scroll down). Fortunately, we haven't came across any apps that _need_ the buffer to be a different size than the viewport, and it's a technically valid console configuration, so apps should have been able to support it before.
Input is also tricky - VT doesn't let you express input with as much fidelity as a console app might be expecting, though this we're working on a solution for :)
Now Windows just needs to ship with a decent pager. :-)
Programmatic access to scroll back is useful for a few things. For example, back when I was on Windows Phone, I wrote a compiler wrapper that would scroll back to the first error message.
It'd be nice for the POSIX terminal world to standardize on similar scrollback access. I know the zsh people would love it.
*nix consoles typically have two buffers - many full-screen temrinal applications switch to the "alternate screen" on start and back to the principal screen on exit. That's why when you exit vim(1), you see the terminal state back as it was before you started it. Will ConPty support this?
Yep! It'll probably act a little different than you'd expect - the pseudoconsole itself will switch between the primary and alt buffers, leaving the terminal in the main buffer always. We'll "render" the contents of the alt buffer to the terminal, then when the client app switches back to the main buffer, conpty will re-render the contents of the main buffer to the terminal.
It's not the most elegant solution, but we're still very early in on this project. We still have lots of improvements to be made to the infrastructure and translation, and even as I type this up, I'm thinking there's probably a better way of handling alt buffers.
Will this replace Command Prompt or Powershell? Or is this more the back-end for these apps? I believe that we are heading for console overload (in a good way?!) with Command Prompt and Powershell installed on every computer and Debian/Ubuntu available on the MS store.
First off: command prompt (cmd.exe) and powershell are commandline client applications. They are shells just the same as bash is.
All commandline clients run attached to a console server, and that server is conhost.exe. Conhost is responsible not only for being the console server, but drawing the actual terminal window these apps run in. So when you alunch cmd or powershell, what you're seeing is conhost.exe "hosting" these console applications.
What we're exposing here is the "master side" of conhost, which will the other applications act as Terminals, like how there is gnome-terminal, xterm, terminator, etc on linux.
Is there any chance cmd and powershell will improve from a user interface perspective? And perhaps become usable? Cmd has been garbage since it's inception.
cmd is parked for pretty much everything except for major issues. It's a scary codebase that has a LOT of code that's dependent on it, and we can't really add any new features there without the possibility of breaking someone.
This feature is mostly focused on the other end of the communication, on being able to create new Terminal windows to run shells inside of them.
Is the pay-off big enough? Who knows what garbage code they might want to hide in there, for both technical reasons (including security) or even legal ones.
Is there a chance this will be connected to a functional shell interface. I get your point that cmd cannot be upgraded because of legacy issues and that is understandable and unfortunate, but windows needs a proper shell. This is obviously a great start for one side of the equation. But until there is a decent terminal app, windows will continue to be a nonstarter.
It sounds like you're asking for two different things here:
cmd.exe is a shell, and that's the guy that's parked.
conhost.exe is a terminal, and that's under active development, though it's slower than something like VsCode, because we can't just go adding features as we see fit, we have a LOT of back compat we still need to support.
Fortunately, conpty will allow for the creation of new terminal applications on Windows. If you're looking for a better shell experience on windows, I can point you to powershell or even [yori](http://www.malsmith.net/yori/), which looks pretty cool
Exactly, I understand there are different underlying concepts and systems to the front end and what it interacts with. It just seem incredible that windows is basically stuck with a windows 95 interface for a shell.
Powershell is open source, the 6.1 preview 4 is nice and fast, and you get real objects with keys rather than scraping for regexs all the time like bash.
Powershell doesn't get any credit for the tab completion when the default behavior for it just makes the verbosity even more of a nuisance. Having tab completion scroll through all the possible completions one at a time doesn't save keystrokes in most cases, especially when there are dozens or hundreds of options that are really long so when you give up on tab completion and decide to type it out manually in full, you have to erase 10-20 characters. The bash-style completion behavior of completing any unambiguous characters then giving you a list of the possibilities would be even more useful for powershell than it is for bash. But Microsoft once again had to throw in gratuitous differences at the cost of usability.
That's subjective opinion. I much prefer the Windows way because I can either specify additional characters or keep hitting tab to until what I want comes up. "Display all 1026 possibilities? (y or n)" certainly isn't an improvement.
Not to mention of course that it tab completes things bash can't and doesn't, and all the other reasons your initial comment was wrong.
> I much prefer the Windows way because I can either specify additional characters or keep hitting tab to until what I want comes up.
You can't specify additional characters until after you've erased all of the incorrect trailing characters that powershell filled in, and that's where powershell's completion method runs up the keystroke count unreasonably. Your comparison isn't valid if you ignore that aspect. It's also not very subjective at all. We're talking about objectively countable keystrokes.
I will concede that out of the box bash doesn't offer completion for anything other than file and command names, but it does include a programmable completion feature and many packages provide completion rules for their commands. It's up to the distro to determine whether to enable all of those completion rules by default or to stick with the more limited but predictable file-only completion behavior.
Aside from that, I can't see what "all the other reasons your initial comment was wrong" are; you appear to have only cited the existence of concise aliases for some commands as a refutation, and that obviously doesn't put powershell ahead of bash in any way, just lessens the severity of that downside.
> you appear to have only cited the existence of concise aliases for some commands as a refutation
You claim that PowerShell is overly verbose, I point out that it is only verbose if you intentionally make it verbose for readability, you proceed to pick nits and pretend you've done some elaborate study on keystroke counts or something, then resort to tired old Linux evangelism mainstays like blaming the distro.
Ah but see, now you're conflating two different things:
cmd.exe is a shell
conhost and cmder are terminals.
I believe cmder can come with git bash as well, which is also a shell.
The confusion comes from when you launch cmd, the window that appears by default is conhost, with cmd running attached to it. When you launch cmder, it's also running attached to cmd.
The back end. Basically people like ConEmu and Hyper and Terminus have been having to use various unreliable hacks for ages because there was no real console API. Now there is one.
It's already available in current insider's builds, and will be landing officially in the next available Windows release some time later this year.
We're still working with ConEmu, VsCode, and OpenSSH to get them all over to the new API, with varying levels of adoption in the next few months likely.
Currently, WSL is also using the same functionality, if you open a WSL distro and run any Windows executables (eg `cmd.exe`), they'll run attached to a conpty. I use this as my daily driver.
Out of curiosity, are you backporting onto Windows 10, or is Windows 10 the only release vehicle for everything in master? If the latter, how are you releasing piecemeal?
Pretty much any new features we release are only available on new Windows 10 releases. Unless there's a gigantic demand for a feature, or business impact, we're not really capable on our team of backporting anything.
What I meant is: are you building a Windows 11, or is Windows 10 the only release vehicle? I'm trying to imagine you pushing to a master branch to integrate into what becomes the insider builds, and then... either those become a future release of Windows 10, or you backport onto a Windows 10 branch. Neither option would be very appealing to me if it was up to me.
Windows 10 is the "last Windows ever". There's a pretty wide tree of branches under "winmain" that teams contribute to, and at each layer more branches are merged together, until they finally hit winmain, and then weekly they tag (more or less) a build from winmain as the Insider's build for the week.
It's actually a lot more elegant than you'd expect it to be for a project with as many developers as Windows has
While we're talking Unixisms, Windows needs a dup2(2). That is, given a HANDLE, you should be able to swap out its backing kernel data structure with that of another HANDLE.
Without this, I/O redirection is slightly broken. Last I checked you can't change where stderr goes after the process starts, for example. [SetStdHandle doesn't do it at the right layer.]
I always liked the Console API where you can set the color of the text without actually changing the text that is written to Stdout. No issues when piping the output somewhere else. No need to check whether the output is getting piped.
That'll work just the same as it always has :) Existing commandline applications won't be affected by this feature, but it will open the doors for an entirely new class of applications.
If I use the System.Process API in .NET and redirect the Stdin/Stdout to a stream inside my application, does the framework spawn an invisible console and scrape the output? Or does this work differently? I always did it that way and thought the 3rd party terminal emulators also do that. Why do these emulators have to do it differently?
I currently use ConEmu + zsh via MSYS2 as my preferred shell on Windows. I need to run many interactive programs like `python`, `node` etc. via winpty, e.g.:
```
alias node='winpty node.cmd'
```
With the new ConPTY, will I be able to run native Windows programs directly? If so, that would be huge, winpty (while I'm really thankful it exists) is a PITA in practice, see e.g. https://github.com/Microsoft/vscode/issues/45693.
Is there any way to get this backported to Windows 7 - or run a W7 userland on top of a W10 kernel? I'm actually serious about this one, I can't stand this semi-"mobile-first", flat UI of newer Windows generations, and the privacy invasions and ads are other hard blockers for me - but that WSL layer or the new console subsystem seem to be pretty nice features.
Functionally, we support all of the VT sequences you'd need to make ncurses work resonably well on windows for a few releases now (ever since WSL was introduced). If you could build an ncurses that assumed TERM=xterm-256color, then you might be able to get it to work on windows.
I sure hope so! I can run cmd.exe inside gnome-terminal running on WSL right now - granted, WSL is doing some magic to make that all work, but it should still work for terminator to do it to.
Do you think that the people who implemented the Windows Console, especially the people working on Windows NT, did not know about Unix? People try different approaches, sometimes they don't work out.
And it's not like Unix is the Word of God, anyway, it has plenty of flaws.
(Yeah, after a long time on internet forums I get kind of touchy after someone copy-pastes the same old and tired line.)
1. They didn't care about command line tools, in many ways (usability) they're a huge regression. After all, it was in the name of the product: Windows.
2. Who says that in-band communication like Unix is doing is necessarily better? See pastejacking and other shenanigans.
The problem is that in this case it's not understanding Microsoft's own prior software that condemns one to reinvent it. Microsoft's second POSIX subsystem for Windows NT, a.k.a. Interix, had all of this.
If Microsoft is in the mood to fix old problems, right ^there you've got another old problem: its bizarre API that is different to everything else. Designed that way to lock everyone into their OS.
In 2018 nobody has the time to learn this. Just use a cross-platform API and if it doesn't run on Windows then just don't run Windows.
As a developer, using Windows for development is against your own best interest. If you like to be treated as a dog that is not allowed inside the house, use Windows.
What cross-platform API would that be? ioctl(tty_fd, TIOCSWINSZ, &size); ? How does the user get the TTY FD? open("/dev/tty0")? Or should they implement SYSV compatibility and use "/dev/vt0"? Or perhaps follow FreeBSD, which has "/dev/ttyv0"?
Honestly, learning Windows is just like learning another programming language. This API is designed to be the Windows equivalent of a unix API - of course it's not going to be the exact same thing, but functionally it does the same thing.
