In general, command-line argument parsers should just follow the GNU style. No more, no less. Deviations confuse users as it is not immediately obvious to them what rules a parser is imposing.
> options can have multiple values: -a 1 2 3 means that a is an array/slice/struct of three numbers of value [1,2,3]
Allowing multiple values is inconsistent because you can't tell in "./cmd -a 1 2 3" whether 2 and 3 are positional arguments or arguments for -a. This is not a GNU style. The GNU way is "./cmd -a 1 -a 2 -a 3" (or "./cmd -a 1,2,3"). This package supports that, which is good.
> option values can be separated by a space, equal sign, or nothing: -a1 -a=1 -a 1 are all equal
"--a=1" is a GNU style but "-a=1" is not. This is a minor issue, though.
Also, does this package support "--"? Everything following "--" should be treated as positional arguments.
I disagree with it being a minor issue. If I write a shell script around a program that accepts GNU-style arguments, I expect the following to be correct:
./cmd -a"$USER_CONTROLLED_DATA"
A program using this package would break that assumption, introducing a bug where this user-controlled data cannot start with an '='.
This syntax is supported by argparse and clap, the most popular argument parsers for Python and Rust respectively, and it seems to have caused almost no problems for them. It's a problem for the uutils implementation of cut, since `cut -d=` is common, but that's the only instance I could find after a long time scouring search engines and bug trackers and asking for examples.
If anyone does know of other examples or other places this has been discussed I'd love to hear it though, maybe I just haven't found them.
(Also, the more reliable way to write this in general is `-a "$USER_CONTROLLED_DATA"`, since that'll behave correctly if $USER_CONTROLLED_DATA is empty. As will `-a="$USER_CONTROLLED_DATA"` if you know the command supports it.)
In the Gentoo world, sometimes you need to give an exact package name which looks like `=net-misc/foo-0.1.2-r1`. The exact match has to start with the '='.
I think short options taking a value in the same argv (i.e. `-o=1` stuff) isn't a GNUism mostly because it's backwards-incompatible with POSIX. `=` is a valid getopt option character, `chmod` uses it.
That said, I think? 'nloomans means for USER_CONTROLLED_DATA to be a set of short flags, not flag values, as in:
root@08e9950d5bfd:/# export USER_CONTROLLED_DATA=lh
root@08e9950d5bfd:/# ls -a"$USER_CONTROLLED_DATA"
total 56K
drwxr-xr-x 1 root root 4.0K Mar 23 16:51 .
drwxr-xr-x 1 root root 4.0K Mar 23 16:51 ..
[...]
Not that I've seen this in the wild before. But everyone's use of bash is a unique personal hell anyway.
Anyway, one other alternative for the `cut` situation is to allow either ':' or '=' to optionally separate the key and the value. Then you can say `cut -d:=` or `cut -d=:` if you wanted to use either one. This is what https://github.com/c-blake/cligen does (for Nim, not Go).
The problem is existing shell scripts and muscle memory and command histories. `cut -d=` has always worked and works on all the other implementations so it should keep working if you switch to uutils.
Literally any short option key that takes a string or a char that could legitimately start with '=' has the problem, though, not just `cut`. The '=' will be "eaten" by one tool and left in another. But you know that. You write a bit as if we disagree, but I don't see any real point of contention. :-) Also, uutils has a very strict "drop-in" agenda. So, as another e.g., if you want `cp -t=foo` or `cp -S=foo` to work the same, you're going to have trouble if that '=' is eaten.
So, in this case that would seem to imply a problem for any utilities with options taking strings-or-chars, not merely `cut -d`. If uutils really wants to be strictly drop-in compatible, they may well need to roll their own option parser or twist the arm of whoever's they use to provide a mode for them.
In the more general case, "cross compatibility" may just always be limited by the reality that people just disagree on this stuff "more than they seem to think they do" (at least in my experience) and definitely more than they wish they did. I surveyed my /usr/bin once and like half of thousands of commands did not work with --help (yes, running that took some confidence in backups! but anyone could replicate on a throwaway VM or something). Consistency is nice, but consistency with what? -l=foo is consistent with --long=foo, but not (some, but not other) historical things.
I'm not sure there will ever will be a world in which you don't need to know which PLang/CL toolkit was used to make a CLI utility if you really want to know its syntax. The article's lib is going its own way from the Go stdlib. POSIX is pretty darn calcified. A 15 year old Python stdlib thing is unlikely to ever change in this regard. Python also allows "--beg" for "--beginning-long-option" if nothing else starts with "--beg" even back in its optparse days and that also tends to be controversial. cligen tools actually provide a --help-syntax. Maybe something like that could take off?
I can think of a lot of cases where it theoretically could be a problem, but `cut -d=` is the only one I've found so far where an end user ran into trouble because of this ambiguity, and I think it's the only one for which uutils bothers implementing a workaround. That's why I give it special attention.
> You write a bit as if we disagree, but I don't see any real point of contention. :-)
The `cut -d:=` spelling solves a different problem than the one I meant (and the one you're now talking about). But we're mostly on the same page!
It is safer to just put in the space (much like you put in the quotes to be safe). Python's argparse will also accept but not require an `=` separator (maybe optparse, too - I haven't checked that one).
It's only "safer" because argp has this particular bug. It's safer for argp (or python's argument parser, for that matter) to not have surprising buggy features like this.
While it may go against your personal expectations, I believe at least the Python variant is intentional not an accidental/unintended "bug".
"Surprise" is observer-relative. From a blank slate, if --long=val and --long val both work, then why not -s=val and -sval and -s val for a short -s?
So, I think the right word here is "disagreement" (perhaps about "which consistency - history vs. internal"), not throwing shade by simply declaring/asserting it a "bug".
Unfortunately a lot of people nowadays don't seem to notice or care and you even get weird arguments like the one about "Go style" vs "GNU style" below. I've also heard arguments about making it more "user friendly" (at the cost of any power user wanting to re-write your tool).
I wouldn't mind if GNU style was consistently extended to allow annotated required arguments. But that's about it.
That being said. As a rule, if your command line utility has enough positional arguments that you're forgetting which one is which, it's a badly designed command line interface. Most often it's because you're doing too much, not settling on sensible defaults for things which you shouldn't need to specify every time, or just doing something outright weird.
IME commands get there by expressing complexity that belongs in a full configuration file on the command line (as well / instead).
It's nice if things are simple enough that a hand-full or two of flags are sufficient. However more complex programs that do more or substantially different things from traditional bytestream filter programs often do need proper configuration files.
Just to clarify, the documentation was a bit outdated, but `./cmd -a 1 2 3` would not mean an array of `[1,2,3]` but exactly as you mentioned: option `a` with two positional arguments. And yes! The `--` is supported :-)
Are you saying this because you think the GNU standard is particularly well thought out, or because it's so common that deviations are confusing in general?
I'm so used to the GNU conventions that I'm not really aware of what the alternatives are or what their merits are.
The most important part is to have a standard, which doesn't need to be perfect. On argument parsing, I actually think the GNU way is the best so far. I have seen various deviations from GNU and I personally regard all of them inferior.
POSIX doesnt even define long opts. I conciously dont follow POSIX these days. The standard will adapt once enough pressure has built up as they mostly document existing things instead of innovate. So I need to apply pressure.
POSIX doesn't even provide any utilities to write long opts and it also doesn't even define any long opts for basically any of its commands, so by default it is barely usable in maintainable scripts.
Except it's not? There are tons of flag packages in real use, because the stdlib one kind of sucks. Even the "go" tool itself works around some of its limitations (things like "go test ./... -v" won't work out of the box, since it will stop parsing flags at the first non-flag, so the go command reorders os.Args before sending it off to the flag package).
Things like "-a 1 2 3" are not standard "Go style" at all. I've never seen that.
It doesn’t. It’s simpler, which makes it not suck in my books.
All the stuff about short vs long, grouping short flags together, allowing the last flag of a group of short flags to take an argument, allowing a flag to take multiple values, etc. is cool, but confusing and quite hard to do right.
You can't expect the user to remember upfront what language the tool they're using was written in and context switch the style they're expressing their intents in based on that. The Go community has consistently shown it's not very good at making well-thought out decisions. GNU is obviously the better approach here.
I think you mean "inconsistent at making good decisions", which one can expect to be quite common, but GP probably meant "consistently making bad decisions", which is different.
I'm implying that a statement pointing at and deriding A for doing X is useless or misleading, if every alternative to A does X, arguably even more so. The precise interpretation of a given X does not matter.
you two should get together and propose a better solution, with a working implementation they could adapt if needed, and integrate.
talk is cheap. (I'm not saying that you're full of hot air, I'm saying that if you want to make Go's flag package better, make a proposal with good rationale behind it, include a good implementation, and see what the community says.)
Go's flag package is covered by the backwards compatibility promise, and no large-scale change is going to be accepted. Pushing proposals to it is a waste of time.
But it doesn't need to be changed. Nothing stops anyone from using any of the several argument parsers for Go. The one included in the standard library is just that and nothing more: included in the standard library. It is not privileged in any manner (beyond that brute fact), it has no access to any sort of internal standard-library only functionality that gives it access to performance external packages can't have, it isn't particularly integrated forcibly into anything else included with Go, it is not mandatory, it is not even something the community deeply believes must be used and anyone who uses alternate mechanisms is violating any sort of nebulous community standard. It's just a package.
This is one of those cases where people often rather casually say "I don't like how Go handles X", or, indeed, any language with a standard library (this is not a Go-specific problem) when they really ought to say something more like "I don't like how the standard library flags package in Go handles X". But it's not the language doing it, it's the library. For a non-Go example that I've encountered in the wild, it isn't really appropriate to say "I don't like Python's GUI", referring to the built-in Tkinter binding. It's just a library. Python's got a dozen other choices, including direct bindings to every major toolkit in every major OS. It's not "Python's GUI", it's just one that ships in the standard library, not the sum total of all of Python's GUI capability.
Could not disagree more. Everything that affects your use of a language is that language - the standard library, the ecosystem, the community, the documentation, the tooling, every single thing.
The standard library is special. It sets the blessed way to do X, e.g. Context, the Writer and Reader interfaces, etc. Claiming that you may as well write your own is like claiming you can just fork Linux if you disagree with its direction. Good luck with that.
Bringing up Python is particularly unfortunate since its packaging story sucks, so whether a package is included in its stdlib or not really matters.
Sure, argv parsing is an isolated piece of functionality, so in that specific case it doesn't really matter what package you use, but the sentiment is incredibly wrong in general.
People are not looking to "add" to the library. They don't like it at all and want to completely stop it from doing what it does. There is no proposal process that is going to make them happy.
Fortunately, that is not a prerequisite for their happiness.
The standard library isn’t going to change because of Gos backwards compatibility promise. Which is a good thing.
People can import their own flag parsing libraries, hence the HN submission we are discussing at the moment.
As for “talk is cheap” comment, I’ve written more open source code for working with software in the command line than most people. And that includes tools that parse other CLI tools and man pages to provide autocompletion hints. But that still doesnt mean that Go should make a breaking change to their standard library based on my personal preferences.
I didn’t say that you should aim to replace the existing package. You imagined that as a requirement.
How did log/slog appear if the Go standard library already has a logger and a backwards compatibility promise? Because you can add stuff without breaking backwards compatibility.
Go’s backwards compatibility promise is good in the short term, but if it has no expiration, that promise will be the rope wrapped around Go’s neck as it is pushed off a small platform and is left to dangle as the crowd watches.
Do you think that backward compatibility promise will still be upheld in 500 years? If you think not, then you agree that there is a line somewhere between 0 and 500 years and that on the other side of that line that promise makes no sense.
What about 50 years? Do you think in 50 years we’ll still have the same flag package in the standard library that we have today? I don’t. I certainly hope we don’t.
The more one clings to that backward compatibility promise, the more the backward compatibility promise strangles you.
I hope we incorporate lessons learned from Go 1 into some new version of Go, whatever its name is. I certainly DO NOT hope that we ignore what we’ve learned and fail to act because of a promise to never abandon the mistakes that have been made.
The original points I make are still just as valid. Your counter, while not technically incorrect, doesn’t change the nature of the problem nor change my mind that a Go proposal is the right way to focus my energy at this point in time. Particularly when I’m already focused on several other open source projects in the same problem domain right now.
Time is a finite resource and there are already other packages out there that satisfy the problem, so why add to the noise?
One could flip the argument and say: if you’re so convinced that a proposal is the right course of action, then why don’t you create one yourself?
> if you’re so convinced that a proposal is the right course of action, then why don’t you create one yourself?
because I'm not annoyed by the existing flag package, and the people I was replying to are, that's why. that's why I suggested that they submit a proposal in the first place.
this is apparently a contest to see who can misunderstand me the fastest and you're all winners in this race.
there is no better person to produce a proposal than the person who best understands why it isn't sufficient or is wrong in one or more ways.
I didn't say it was easy to guide a proposal from the proposal stage through to acceptance, or that I was optimistic about doing it. (you think I am misunderstanding you?) I merely feel like the people who like something the least are probably the most qualified to produce a quality proposal which has a chance at improving the landscape. that's all.
By the way, if i haven't said something outright, I did not infer that thing. Don't read between my lines, there's nothing there.
I can use a different package so what’s there to be annoyed about?
> there is no better person to produce a proposal than the person who best understands why it isn't sufficient or is wrong in one or more ways.
Dislike != insufficient
It’s a personal preference not a technical criticism.
> By the way, if i haven't said something outright, I did not infer that thing. Don't read between my lines, there's nothing there.
If you’re going to play the “you’ve misunderstood me” card then at least make sure you’ve understood people correctly yourself.
> I didn't say it was easy to guide a proposal from the proposal stage through to acceptance, or that I was optimistic about doing it
Changing the way the flag package would need to be changed to make myself (and others) happy couldn’t be done in a backwards compatible way. Thus it would have to be part of Go v2.0, which could be a decade away or might never happen.
And since all that the change would bring is an alternative, more GNU, way of parsing flags, it makes this hypothetical proposal a really low bar for improvement (unlike your other examples) thus this hypothetical proposal is unlikely to ever approved even if a hypothetical v2 of Go were to ever be released.
The whole premise of this changing is so remote that even suggesting I write a proposal in the first place is a massively optimistic take.
And for the record, I fully support Go not changing their behaviour here. Keeping the flags package as-is is the right thing for Go to do. So I wouldn’t even support my hypothetical proposal here even though I don’t personally like the existing flags package.
This might sound too nuanced for HN, but you can dislike something while still supporting it’s existence. Change here, in my opinion, would be worse than the status quo.
When I use a command-line tool, I don't know and don't care if it is written in Go or not. I just want to use it the same way as most of the other traditional Unix tools. The Go style gets in the way. We would have much more consistent CLI between tools if Go had just followed the GNU style at the beginning. The same can be said to many other languages like Nim that want to reinvent command-line argument parsing with their standard libraries. https://xkcd.com/927/ came to mind.
For, like, 95% of the tools out there, I’m going to use --long every single time.
Anyway, I don’t even know the options that a program takes until I read the program docs. The program docs will tell me that the option is -host=localhost or --host=localhost or --bind-addr=localhost:8000.
The other 5% are tools like ls, cp, mv. As far as I care, ancient tools are the only ones permitted to have short options that combine into a single option, like old-school getopt. Maybe a few exceptions now and then.
I use short options when I'm typing at the command line,
but for scripts I prefer the long options by a wide margin.
It's just too painful to come back to script after some time and see a string of short options that looks like line noise barfed out of a 300 baud modem.
Cobra https://github.com/spf13/cobra uses https://github.com/spf13/pflag, which supports GNU style flags. Cobra has been developed for many years now and has a ton of additional features, like automatically generating the autocompletions and has tons of unit tests.
Cobra is certainly popular, but it has many weird edge cases and limitations. It's been a while since I've used it so I can't recall the specifics, but I do remember it being a very poor experience, both as a developer and as a user.
I have used both argp and Kong.
I consider them two of the best Go CLI libraries for different reasons.
Kong has a full set of advanced functionality like flag groups and associating environment variables with options.
It is highly useful in a larger project with a complex CLI.
For a simple command-line interface, I prefer argp's small API and defaults, like exit with status 2 on bad usage.
(Disclosure: I contributed this default to argp.)
Kong 1.7 has a disadvantage when it comes to negative numbers.
argp parses `foo --bar -1` as `foo --bar=-1` the way you expect when `--bar` takes a value, but Kong treats `-1` as a flag.
They all have their warts, but "just pick cobra/kong/cli" is what I go with these days. I'm still partial to Cobra because of the nice autocompletion scripts it generates for me so I get tab-completion for free.
one thing I liked about cobra is the auto complete support, but I don't like the manual process of define command and parameter and not easy to extend, so I wrote https://github.com/hujun-open/myflags, it is built-on top of cobra but allow uses a struct to define all commands/args and also easily to extend support new types (even existing types)
I’ll second that. Cobra is the industrial strength flag parsing package most of the professional Go ecosystem has settled on. It’s the boring choice in the best sense of the word. Use it and move on with life.
I've become a fan of https://github.com/jessevdk/go-flags (also enjoyed structopt which is similar for Rust). I much prefer defining the flags statically on a type which stores them after rather than building it all up programmatically.
Otherwise, I'm not really sure what this offers that alternatives don't, apart from the -a 1 2 3 -> [1,2,3] business which seems highly confusing and undesirable to me so I'd think of it as more of an anti-feature.
Also a fan of go-flags, I use it by default most of the time (unless the use case is so basic that the stdlib flags usage would be sufficiently simple).
I think my only real point of "frustration" (mild at best) is that flag composability isn't always possible (eg embedding/sharing common non-root flags across flag structs), IIRC struct tag eval doesn't reach nested types.
Other than that though, it's one of those nice libraries that just does a thing and gets out of your way.
Hmmmm I thought that sharing a common type did work in the way you would expect it to? I haven't tried embedding with it though I don't think.
TBH I rarely do it much anyway, mostly the flag struct ends up being one anonymous struct which is operated on in the main package only. I do like writing custom types for parsing though - I've seen engineers do a bunch of post-hoc parsing logic which generally just ends up being messy.
encoding/json does not make liberal use of struct tags, it uses only one. It's also battle tested and lives in the standard library. I don't have faith in many other projects to get bug-prone reflection code right.
It does have a microformat within that single tag though ("x,omitempty" etc) so there is a bit more complexity than 'just one tag' indicates.
Anyway, I think you're overstating how hard reflection-based code is. Yes, it is tricky and should mostly be avoided, but it's not beyond the scope of human intellect to understand (especially Go's which is pretty sensible).
Reflection is slow and bug prone. Struct tags are widely considered in the Go community to be a nasty hack yet a necessary evil. They should be used at little as possible because if you mess them up you don't get a compile error, you get a runtime error.
Beware of microbenchmarks, but in my tests the above package only added around 0.001 ms over the standard library flag package. Even if real world usage increases that by several orders of magnitude, it isn't going to matter. No human will ever notice.
I know of UNIX parameters, with a single hyphen & a single letter, and GNU parameters, with a double parameter and a word. But a single hyphen and then a word: *mind broken*.
When I first saw that on Terraform, I was upset & conflicted.
OK, I know that Java also has mental parameter options like that ('-version', note the *single* hyphen followed by, gasp, a word!), but I just dismissed it as weird enterprise-ism.
Then I saw that this 'single hyphen + word' was an accepted convention for parameters, in the Go world & I was disappointed.
Glad that this package is fighting the good fight.
> OK, I know that Java also has mental parameter options like that ('-version', note the *single* hyphen followed by, gasp, a word!), but I just dismissed it as weird enterprise-ism.
I think "mental" is being too kind about java.
$ java -version
and
$ java --version
both work, their outputs are different, and `-version` goes to STDERR whereas `--version` goes to STDOUT.
`find` is also notorious for its parameter style. But in my opinion programmers sometimes put too much importance on following conventions. You're not gonna go blind from looking at an incorrectly indented line. If the founding fathers of unix didn't want us creating horrifying monstrosities, they shouldn't have given us unlimited access to argv :)
Aside from the other comments, an ick I have is that struct tags are being abused here. Instead of providing a grouping like json, yaml, mapstructure, gorm, etc - it just goes willy nilly with struct tags like it owns the whole space. I’d like to see it grouped like…
type Conf struct {
Thing string `argp:”name:thing;short:t;desc:A thing you can do”`
}
Maybe the README should mention whether reflection (the part that would disable dead-code elimnation) is used? That was a pain point for me when I had to investigate a binary bloat issue due to reflect.
I might be the only person in this camp, but I find the "standard" command line arguments style absolutely repulsive. I write tons of CL code, and I always use easy key=value notation (sometimes it's flag=true, which I consider to be a minor sacrifice of conciseness in favor of consistency and readability).
It is more important to follow a common convention than agree on which looks better. If you are lucky you get used to common conventions and spend no time annoyed with them. If you are unlucky you will struggle to let it go. Struggling to let go is a disadvantage.
People who ignore convention and insist on whatever they like are never any fun to work with.
According to the example and judging by the code at a quick glance, it does not, it seems to exit, which is unfortunate. Perhaps the author may correct me.
> options can have multiple values: -a 1 2 3 means that a is an array/slice/struct of three numbers of value [1,2,3]
Allowing multiple values is inconsistent because you can't tell in "./cmd -a 1 2 3" whether 2 and 3 are positional arguments or arguments for -a. This is not a GNU style. The GNU way is "./cmd -a 1 -a 2 -a 3" (or "./cmd -a 1,2,3"). This package supports that, which is good.
> option values can be separated by a space, equal sign, or nothing: -a1 -a=1 -a 1 are all equal
"--a=1" is a GNU style but "-a=1" is not. This is a minor issue, though.
Also, does this package support "--"? Everything following "--" should be treated as positional arguments.