Hacker News new | past | comments | ask | show | jobs | submit login
Writing Small CLI Programs in Common Lisp (stevelosh.com)
174 points by reikonomusha on March 17, 2021 | hide | past | favorite | 61 comments



Common Lisp is awesome: image-based development, compile-time type checking, self-contained executables, stability…

https://github.com/CodyReichert/awesome-cl/

https://lispcookbook.github.io/cl-cookbook/editor-support.ht... (Atom, Sublime, VSCode, Jupyter…)

https://lisp-lang.org/success/ (and https://github.com/azzamsa/awesome-lisp-companies)


For those unfamiliar with Common Lisp there is also Babashka which allows for Clojure shell scripting without the JVM startup time.

https://github.com/babashka/babashka


Common Lisp itself has Roswell, which I am disappointed to see is not even mentioned in the article.

https://github.com/roswell/roswell


It's sort of mentioned indirectly at the beginning in that source-code scripts that use a shebang, or make use of exit, can introduce development barriers. I've not written many CL scripts but I like Roswell too, and haven't found those barriers significant/odd to work around. I agree Roswell is definitely worth checking out. (Plus at least for me it gets even more use by letting me trivially run my tests with alternate CL implementations...)

Like, I didn't even consider including "some ugly reader macros" to handle the shebang issue, since my approach has been to comment it out while developing and uncomment once I was ready to use it as a script. But I think this simple macro I put just now in my $HOME/.sbclrc file should suffice:

    (set-dispatch-macro-character #\# #\! (lambda (stream subchar arg)
                                            (declare (ignore subchar arg))
                                            (read-line stream nil (values) t)
                                            (values)))
And in some programs (not scripts) where I make use of uiop:quit, I just don't call those paths during dev, but this is the same approach the article has differentiating calling run vs toplevel.


About uiop:quit, that sometimes are in the middle of a script: lately I have wrapped it around a check of the nature of the terminal: if the terminal is dumb ($TERM), we are very likely inside Emacs/Slime, so we don't quit. https://github.com/vindarel/termp


A quote from that link:

> Life's too short to remember how to write Bash code.

Hard agree.


I really feel bad that I don't write more Bash, but every time I learn some, it's gone. I cannot retain that language, and I don't understand why: I'm a language gourmand, in general. But I throw Python, Scheme, or even Nim at little scripts every time, because I have to look everything up if I do it in Bash.


I've always internally thought that there are some languages that our brains just _don't_ like.

I used to think that there was "something wrong" with me for finding C++ verbose and C expressively dangerous, or to find myself reaching for multi-line awk, sed, and bash scripts that did _stupid_ things when I "knew" that a better solution existed but I didn't know how to _express_ that solution concisely.

Perhaps a more concrete example would be this: I don't use python often and some of its syntactical quirks get me every time -- like, for example, None as a method to insert a new axis by convention in numpy (so much so that numpy define np.newaxis as an alias to it). It makes total sense but at the same time, the fact that the language likes having what almost looks like a C NULL as a method to insert a dimension seems...wrong, like it should be a syntax error.

I've now just come to realise that hey: if you can't express this concept well in one language, and you _can_ in another, it's well worth prototyping the solution in the right domain and one could always write something in a "sensible" language later if it turns out to work.


I realized at one point that I was writing an awful lot of shell code despite never really learning shell.

The only way I found to learn it that was fun was too write a shell implementation myself. Just POSIX, though so I never learned ksh/bash features.


That's a really great idea, and I expect to get nothing done for the next week thanks to you.



Every time I've started to write a shell script, I've finished it in Python.


Or nodejs. It's python, but better.


Life’s too short to not learn how to write bash code.


I'm astonished that we haven't moved past bash.


I'm not. "Everything is a string" is an incredibly powerful worldview for tying systems together where those systems never put any thought into interoperability. Any realistic bash alternative would either need to rewrite the entire user land with a more structured exchange format like JSON or .NET objects, or ship with adapters for most every command line tool we use.


I love writing bash!


Judging from the logo, the name is derived from the Russian word Babushka, which stands for a granny.


Indeed, it's a lovely name


Nim [1] has cligen [2]. An output binary size for the trivial:

    proc f = discard
    import cligen
    dispatch f
is about 708 KiB on Linux, 621 KiB stripped. And if you like colors, hldiff [3] is not a terrible example.

[1] https://nim-lang.org/

[2] https://github.com/c-blake/cligen

[3] https://github.com/c-blake/hldiff


Nice writeup! I have fairly recently written CLI programs in Common Lisp, GAMBIT Scheme, and Swift. All good experiences. To be transparent, I don't write bash scripts more than a few lines. For something short, I would prefer Ruby.

EDIT: in the last month or two, I have experimented with Babashka (mentioned in this thread) also.


I wish they'd mention the size of the binaries produced this way, and whether they are standalone.


The following data is for standalone executables (modulo things like libc and libm that most Linux executables depend on)

Columns are:

- apparent size (I run FS compression so the actual on-disk size is nearly the same between the compressed and uncompressed)

- Executable name: $IMPLEMENTATION.{big|compress} implementation here is either SBCL or CCL. "Big" means I load the drakma library before saving the image (which pulls in things like unicode tables). "compress" means I attempt to enable image compression; the numbers make it looks my system's CCL does not support compression.

- Mean time to run a program that quits immediately as a startup time proxy; sbcl pays a big penalty for compression as without compression it just MMAPs the image in so doesn't ever load most of the image when you quit immediately.

/bin/true is included for a minimal "fast and tiny C program" as a comparison.

    31K /bin/true 85% .0004
    43M sbcl 99% .0036
    29M ccl 93% .0053
    58M sbcl.big 99% .0049
    14M sbcl.compress 99% .2012
    44M ccl.big 95% .0071
    44M ccl.compress 95% .0071


/bin/true will still forever irk me: https://twitter.com/rob_pike/status/966896123548872705


I should also comment that ECL makes very compact binaries, and the runtime, which is a dynamic dependency is a single digit number of megabytes.

The start up time suffers though; on the order of half a second.


Janet is a good option if you are looking for small binaries. Some people argue Janet is not really a lisp. Regardless, it has many lisp features you would expect like macros, is cross platform, and can produce statically linked binaries.


> Some people argue Janet is not really a lisp.

Out of curiosity: on what basis?

Janet is definitely a Lisp, although it's not the (Common) Lisp.


The arguments I have seen are based on Janet using arrays/tuples rather than cons cells. Here is the author addressing this on reddit a while back. https://old.reddit.com/r/programming/comments/aqwedz/janet_i...

The debate continues in the thread. Either way, I think Janet is very useful for situations where you want something lisp like and also want/need small executables. I've experimented with it quite a bit and have found it really useful for putting together cli apps. The sh package is really useful for gluing together other shell programs. https://github.com/andrewchambers/janet-sh


I don't understand the rationale of people tying the state of being a "real Lisp" with cons cells. As if that were the only contribution lisp has made, instead of homoiconicity, REPL-focus, etc.

The polymorphic sequence abstraction in Clojure is the definitively modern way to manipulate data structures in a lisp family language nowadays. It retains `first` and `rest` as abstracted `car` and `cdr`, but in return all collections (vectors, lists, sets, and hashtables) work transparently with it.


What a waste of time... It's like the "smug lisp weenies" of ages past somehow slipped into the modern era.

Janet is a nice embeddable Lisp (that is: it belongs to the Lisp family of languages). I've been meaning to play with it, but concluded that the lack of (robust) stdlib would make it hard to use for standalone applications. Was it a problem for you in practice, or is the small stdlib enough for CLI apps?


I do wish there was an http client/server component in the stdlib. That's probably my biggest gripe at the moment. I think things are evolving in the right direction though. Recent updates have added an event loop module and have improved the net module. I can see a future where Janet becomes very useful for network programming. I think http in stdlib plus some growth in the third party package ecosystem will make Janet more viable.

I mostly stick to using it as a gluing tool for shell scripts for now. So for my purposes it was fine for CLI apps assuming you are gluing together other cli apps. I could have avoided some gluing if Janet had it's own http client. I spent a while trying to dig into some of the packages and to be honest they just aren't there yet. I really wanted to experiment with building a desktop GUI in Janet but I found that it was too much effort to get started. Cljfx (clojure) and racket/gui both work out of the box with minimal effort.


"I mostly stick to using it as a gluing tool for shell scripts for now." You might like https://github.com/babashka/babashka then.


I've been using Janet quite a bit for small CLI apps.

The stdlib is quite enough for basic CLI apps, especially when you use PEGs.

Janet's event loop also makes it very handy for issuing a lot of subprocess calls.

Janet has basic HTTP servers and clients, and there is an effort to shore it up there.


Many people argue that Scheme is not a lisp, so the bar for "some people argue that X is not really a lisp" is fairly low.


I mostly add that some people argue it is not a lisp because frequently someone will comment saying Janet is not a lisp whenever I mention it in the context of other lisps.


Some languages that have CONS cells: Common Lisp, Emacs Lisp, Scheme, Dylan, Maclisp, Interlisp, AutoLisp, Lisp Machine Lisp, EuLisp, ISLisp, *Lisp, LeLisp, Lisp 1.5.

Some languages that don't have CONS cells: Clojure, Janet, Julia, R.

Of course, that is only one attribute, and may not be the most important one, but it's an interesting separating line.

If you describe each language by a set of language feature attributes, you can perform some easy computations (e.g., Jaccard distance, or some clustering) to get more clue about how related these languages really are. I think Janet wouldn't fare so bad in relation to obvious Lisps, so I'm not strongly opposed to calling it a Lisp.


> Some languages that have CONS cells:

Erlang and Prolog also use cons cells... Prolog is also homoiconic. I don't think I ever heard anyone calling Prolog a Lisp though.

Also, Clojure has cons cells. '(3 . 4) is a valid expression in Clojure.

Basically, this is an arbitrary, and rather unhelpful, classification. I don't find it interesting, but to each their own, I guess.


> '(3 . 4) is a valid expression in Clojure

True, just not a cons cell. It's a persistentList with three elements: 3, ., and 4.

Clojure has a higher-level basic data-structure than the usual Lisp. In those Lisps cons cells are are nothing more than a two-element record, a few basic operations (CONS, CONSP, CAR, CDR, RPLACA, RPLACD) and a special data syntax for them: ( a . b ). In a typical Lisp, the dot is not a list element, but divides the car from the cdr. Additional there is simpler notation for (a . (b . nil)) in the form of (a b) .


I stand corrected. I used Clojure briefly a few years ago, so I don't remember much. I even ran a REPL and checked if it's a valid syntax, but it didn't occur to me to check its type. Sorry!


R also has cons cells, called pairlists. They are generally only used for function arguments though.


> I don't think I ever heard anyone calling Prolog a Lisp though.

I guess we can agree that it's not a sufficient condition, but from my selection of languages you may consider that maybe I meant it as a necessary condition.

Another interesting separator is: does it include "Lisp" in its name? Scheme and Dylan don't. Maybe people don't use "Lisp" in the name to indicate that they intend the language to break away in some way from the Lisp tradition. In a similar way, Racket broke away from Scheme tradition, for example.

Kent Pitman, in his "Lambda the Ultimate Political Party" article, wrote that people have been writing programs to help transition code written in the older dialects to the new dialects. The fact that it's doable without expending too much energy also suggests close relationship. Some months ago I wrote some <100 lines of code, consisting mostly in a bunch of macros, to make a program using Flavors work on Common Lisp with CLOS. Are the people using Clojure, Janet, etc. doing this? Can they pick up an old program written in another Lisp dialect that's close to them and, with not too much fuss, make it run? Which dialect would that be?


  user=> (rest '(3 . 4))
  (. 4)

  user=> (second '(3 . 4))
  .

  user=> (nth '(3 . 4) 2)
  4


Yes, I was wrong, my bad :)


I was wondering about that, too. Last time I checked SBCL binaries weren't exactly small, more around 40MB.


Typically; if you use the :compress feature, more like 14 meg.


Another alternative for shell scripting in CL there is https://www.cliki.net/cl-launch


I've recently been using Python3 + argparse for small single file utilities. Used to use bash but have probably converted now with how easy calling external programs is (compared to something like Go)

I'd have a hard time selecting LISP for anything others would use (which is almost all the software I write) unless I worked in a company that was using LISP regularly.


Consider trying a Lisp-like scripting language such as Guile. Back in the late 90s I found Perl reprehensible and avoided it unless I had to use it. Guile became my Perl. I managed entire web sites with it.


I read this entire thread thinking - this is neat and all but it’s a handful of lines of almost english language in a .py file.

Click is a neat upgrade from argparse if you’re ever tempted (assuming something like “pip install —-user” is viable in your situation which isn’t true for everyone).

I’ve seen, but haven’t used https://typer.tiangolo.com/ - i have used his FastAPI and thought that was nicely done (even if python async is a bit annoying to test).

Talking of python cli libs I haven’t tried but will do one day - rich: https://github.com/willmcgugan/rich


Well, but then you'd be writing python. (and having to maintain python runtime environment, which tends to be much less stable than CL)


I try to avoid pip and dependencies for these python scripts. This goal has it's limits.

Click is nice if you want a command with multiple sub commands. I use Go + Cobra at that point via https://github.com/hofstadter-io/hofmod-cli

The human understanding is a big reason why python has gained my favor over bash


I recently switched to Click from argparse and like it better so far.


A lot of these are multi use local run, in CI, and at onsite w/o internet, so we try hard to avoid dependencies and keep them small and pointed.

Once we need deps we need a more complicated setup, so Docker or Go usually prevail


You can always use something like PyInstaller or freeze to produce a single-file executable that bundles all of its dependencies, though at that point you may as well just use Go which does this more naturally. Single-file Python executables tend to be huge and slow to start up, not a big deal for non-interactive server processes, but miserable for CLIs.


I would say: don't do it. Nowadays I'd rather use Common Lisp as a glue for other scripts rather than invoking it along with other programs with shell scripts.

Plus the need for scripts diminues when you can write a function on typed objects (all scripts are parsers).

I also try to seek other IPC mechanisms when possible, such as talking to daemons through DBus or JSON-RPC depending on what is available.

edit: this story is currently on the front page of HN too, https://news.ycombinator.com/item?id=26491858 (Do not use redirection characters in your shell prompt), the neverending examples of how shell scripting makes your life miserable (sure it's helpful, but can't we have nicer things?)


Does this work on Windows? If i recall, there were issues with lisp interfacing with windows (but uiop may have solved it already - I have been to get my programs to work in Windows without issue)


You probably won't be using a Makefile and shell-scripts on windows for doing the building, but yes it works fine.

SBCL had scary "windows not supported" messages for a long time, but that was mainly because none of the core sbcl team had windows machines, so any bugs that didn't reproduce under wine weren't going to get support from them. LispWorks and Allegro both have had first-class support for windows for decades.

As far as UIOP: UIOP provides portability interfaces that make it less likely that something developed on linux will be DOA on windows, as well as smoothing out differences between implementations. If were developing on a single implementation for just windows nothing in UIOP is really necessary (but portability is always a "nice to have" feature that UIOP does provide).


Thanks


LispWorks and Allegro work just fine on Windows.


In Allegro Common Lisp it's

    #! /path/to/mlisp-or-alisp -#!

    ...lisp code...


There is also scsh, the Scheme shell.




Join us for AI Startup School this June 16-17 in San Francisco!

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: