> The ‘Unix philosophy’ originated with Ken Thompson's early meditations on how to design a small but capable operating system with a clean service interface.
Except that a program-to-program interface based on formatting and parsing text is anything but clean.
I thought the same but what are the alternatives? Use a format like json? What happens when the format can't express what I want? The other issue is the output is now deeply intertwined with the program because it's now an API, although the same could be said about text output.
PowerShell gives a good alternative: the things passed between the programs should be objects. It requires, of course, that everything is communicating with a common runtime, but for a shell that is not the biggest burden.
Semi-rigid, informally specified output meant for easy inspection by humans. The output of `ifconfig -a` for example. (That's the worst offender, most other commands have a better-formatted output.)
As a general rule of thumb: if you have a command pipeline that would break because programs in it handle locale differently, it's most definitely text-based program.
I like the idea of Power shell: programs pipe formatted (binary) objects between them, and there exist commands to print them out in a nicely formatted way to the terminal. But it's objects. I can extract information through named fields instead of through magic indexes and regexpes in awk/sed/perl oneliners.
IOW, proper object (meta)model, not sucky text streams. Text is great for long-term storage, but awful for data in flight. UNIX ignored this distinction and decided to use text for everything.
Maintaining IPC/RPC as text has the significant advantage of keeping the programs involved honest, approachable, and easier to debug. Just being able to see the interface by running the program (or reading the config file) makes it a lot easier to learn[1], and bugs in text formats can often be seen visually (or marked in an editor).
The alternative - binary structures - require complex data definitions where you have to care about stuff like exact byte lengths when you have to convert between big/little endian. Some people like to claim that binary is faster, but your bottleneck is not going to be strtol(3). As for advantages in parsing - you still have to parse the binary structure you just read. I suspect most people that think "parsing text" is difficult are confusing the parser with lexer; the latter is the only part that changes when you switch between text and binary formats.
In the long run the initial investment in a text interface can be cheaper than wasting time debugging binary structures. Even more important when thinking long-term: it is a lot easier to inspect a text interface from the outside without the consent or availability of the original author. If you have an old program binary that was used in production for years and no source code or documentation, which would you rather try to debug? An opaque binary file? Or something in JSON or INI format?
[1] however, this is not an excuse to skip proper documentation
"Text" in a computer is bytes + metadata about those bytes. Without that metadata, it's just bytes. Exactly that is what unix pipes are: a streams of bytes, not streams of text.
Ironically, the tools that are happiest about these streams are of course those tools that don't care about text but just stream bytes. The pain occurs when the tools actually need to parse the text.
I would agree much more with the unix design, and think it was a much better implementation of the unix philosophy, if the glue layers were strictly specified or demanded that programs could negotiate things like encodings.
Somewhere between "just bytes" and "binary structures" there has to be a reasonable simple format for program communication. Structured text, or text with a small metadata header (like BOM, but done right) for example.
This is why you the predefined character classes in regular expressions. For example, setting LC_CTYPE changes the meaning of '[[:alpha:]]' and sort(1) respects LC_COLLATE. A lot of work has already been done to solve these problems.
> A lot of work has already been done to solve these problems.
Not nearly enough. You write a script which expects a 'file' at some place, but it will break in french locale because 'file' will be replaced with something like 'fichier'. Sure, you can run under C locale, but then some other script, written for french locale will break. Not to mention date formatting, number formatting, etc.
In comparison, binary or XML (!) become very attractive.
You really shouldn't ever run under the C locale unless your need to support older (pre-locale/unicode) software or data of the same vintage. Doing so defeats the entire purpose.
You're confusing issues here: locale support lets you parse text. Your "file" example is not relevant, and would be part of something like gettext(3). Yes, gettext can be configured from the locale, but that is a separate feature from what I was talking about.
The locale support is how you automatically handle lexing the input stream, which is why I brought up the character classes that, unfortunately, most people seem to ignore, resulting in broken text support.
If you properly support locales, you program will support the user with LC_ALL="fr_FR.utf8" typing their floats as 3,14 automatically.
The fact that your program expect a field name to be 'file' is unrelated, but is something the user could learn by 1) reading your man page, 2) reading your text output and copying it (if appropriate), or 3) reading your error messages that you should be generating when you see 'fichier="foo.txt"' when you were expecting 'file="foo.txt"'. Note: if you used [:alpha:] to lex the input, you would automatically be able to extract the incorrect word for your error message when a user in LC_CTYPE="jp_JP" enters 'ファイル="foo.txt"'.
I believe the problem here is related to a confusion of lexing with parsing. The various locale features solve the lexing problem, but you still have a parsing problem no matter the syntax of how the data is serialized. You have to check that /[[:alpha:]]+/ was 'file' not 'fichier' or 'ファイル', just the same as you would have to check the output of your XML parser that the <file> tag was not <fichier> or <ファイル>, just the same as you would have to check that a binary field was 0x0003 or whatever the flag value was.
You may be looking for a way to automatically discover the semantics (schema) that another program expect - and that would be a nice feature - but that is generally orthogonal to the syntax used for IPC. (must like how XML can be converted to YAML/JSON/BSON/etc). It is also a far more complicated feature, and I'm not sure it can ever really be solved (halting problem, possibly), but maybe a "good enough" solution could be created.
Fully agree with you here. Wanted to also add that even the 'parsing' of text is easy by using standard tools like awk, sed etc.. rather than writing custom code to debug binary protocols. In addition with protocols like JSON, it is easily extensible without worrying about where new data fields are added as long as backward compatibility is maintained - i.e. if designed correctly, the server can be upgraded to accept new fields while still working with older clients.
Except that a program-to-program interface based on formatting and parsing text is anything but clean.