Funny, I did quite a deep dive on this same issue about two years back, and came to exactly the opposite conclusion: keep ASDF, and lean on Nix to do pinning. I now maintain my own Common Lisp package repository, testing daily against nightly SBCL etc, and I've learned that a lot of CL stability is just because test suites are never run. Nix goes a long way to shield me from the haphazard breakages that other commenters mentioned, but of course it comes with a huge downside: you need to use Nix :)
At the risk of derailing the conversation (although Guix is a lisp so maybe not): I agree 100% but also maybe Nix's pragmatism is why it's more popular? "Pragmatism" being a programming language euphemism for "untyped hacky mess".
Why is this in past tense? Guix development is active. It's just Nix is more popular and it's understandable since it's the original idea and it's older.
Guix even has a very active, high-quality blog where maintainers detail major technical accomplishments, long-term goals, etc.: https://guix.gnu.org/en/blog/
From here it seems like they're growing and advancing well. I wish I could find ready historical data on the numbers of packages and services from, say, 4 years ago vs. today, though. I could have sworn Repology used to show year over year stuff but I can't find it now.
This is great news. The updates to SBCL broke some libraries (that is, sbcl got more thorough in finding bugs that some libraries had) and I've had to load patches in for them because of the quicklisp limitation in a patch file. This will make life much simpler as do all of Fukamachi's libraries in general. (His internet suite is fantastic)
Seems like an attempt to bring "lockfiles" or related concepts to Quicklisp. This is a desperately needed quality-of-life improvement for "modern" Common Lisp development.
However, I wish people in the Common Lisp community weren't weary of using ASDF directly. It already supports versioning and many features that do not exist in Quicklisp -- despite the fact Quicklisp wraps ASDF -- such as a package that depends on a version of a library newer (or older) than what exists in the current dist.
It's kind of impressive that Common Lisp is the only language I use where I can't expect code written today to work next month due to Quicklisp lacking versioning outside of the date attached to dists.
faré the guy who briefly took over asdf development before quitting common lisp entirely to do scheme, or whatever he's doing right now, spent significant amount of time turning asdf from a 10kb file into 3000kb file in order to solve problems that only he himself cared about, or possibly only google cared about because faré worked for google at the time (building portability library that ignored and force-superseded existing solutions, and trying to eliminate corners cases of SAT solving). there was a period where he was harrassing (this is a subjective interpretation of the events, but this is my side) implementors to incorporate his recent asdf changes, that were breaking left and right to the point that multiple common lisp maintainers of multiple common lisps were considering pulling asdf from their repos entirely. things have stabilized, because faré quit and there's no longer volatility. we're at a point where there's just no concensus about relying on asdf for anything, and while some people go all in on 30% of available functionality (as opposed to customary 5%), equally many people, the majority, use it as a glorified (load)(load)(load). this is an unfortunate side effect of activist developers in small communities: they can have outsized long term effect on how things are done.
In the Meta-CVS project, I carried a copy of asdf.lisp in which the package was renamed to zxcv, to isolate from the ASDF churn and be able to run my version of ASDF possibly with my fixes even in a Lisp that has ASDF preloaded.
ASDF is the only build system I know which broke on me when I tried to fool it with a link farm (lndir command) into building in a separate directory. It resolved the paths using the truename function which eliminates symbolic links, and then calculated the .fas file names from the canonicalized paths, thereby depositing compiled files into the original directory rather than in the symbolically linked tree.
ASDF also depends on symlinks for external organization. .asf files are often symlinks to the real location, and without these symlinks, things don't work.
i have a patched version of asdf 1.369 which was maintained by gary king, with contributions from luminaries like nikodemus, chrodes, weitz, pvaneynd and kevin rosenburg. i've patched it to support the recent extensions that people seem to like (extra parameters to defsystem, like author). you can drop it into your implementation and quicklisp and it just works. asdf 2, asdf 3 happened since, they solve some kind of problems, i guess whatever google needed to do with its fleet, or maybe it's "easier to maintain", but it's not the kind of problems that surface in like 60% of quicklisp packages.
Could you distribute it? I'm not a huge fan of ASDF complexity and have had many headaches trying to make some custom components work. I would welcome any simple alternative, even though I will probably end up writing my own.
> It's kind of impressive that Common Lisp is the only language I use where I can't expect code written today to work next month due to Quicklisp lacking versioning outside of the date attached to dists.
I think the lack of version pinning is a positive force in stability: reducing the pain of breaking changes increases the likelihood that library authors will break things.
This seems to be the widely reported experience from people who develop in Common Lisp daily or very frequently, and even in my haphazard development I find things pretty stable. And it's not like version pinning is all that helpful in a living project anyway, because you often want to update things, and I notice my JavaScript-using friends constantly suffering from breakages every time they try.
With Lisp, I setup my libraries in jenkins to pull the latest SBCL and quicklisp distro once a month and run tests, it's pretty rare for things to break. Once or twice something with an SBCL release itself, once with a bug involving the interplay of two of Fukamachi's libraries. The SBCL one was noticed by others and fixed pretty quickly, for the library one I was able to use git-bisect to easily find the breaking commit and apply a local fix while I waited for the issue I filed to get addressed. This was during Fukamachi's life-threatening illness period, even, but an alternative maintainer did address the bug on their side. The only annoyance these days is Xach is suffering from who knows what preventing the quicklisp distro from updating since October, and there hasn't been communication with people offering to help, so it's forcing everyone to pull projects in locally (and qlot does help with that) for updates or use an alternative.
The only time I've had CL code break due to lack of versioning was when I was depending on stuff written by the author of Qlot, so I either avoid or locally pin stuff written by him.
Most of my dependencies are really quite stable.
[edit]
I thought of a place where things broke that aren't related to Fukamachi; sometimes foreign libraries have changed to the point where they won't grovel any more, but I don't think that's something that Qlot aims to solve.
Honestly, I just don't think it bothers people much.
When I need a specific version of a library, I clone it to my local projects directory and checkout the version I need. Problem solved. Quicklisp is just downloading a snapshot of the upstream repo anyway.
For me, if I could change anything about QuickLisp, I'd add something like Python's `virutalenv`. I suspect I can hack something up by messing ASDF's source registries, but so far I haven't wanted it bad enough to implement it.
That's interesting because I find it to be one of the most ergonomic build tools I've ever worked with. It sits somewhere between make and autotools in terms of what it does and I find it far easier to use than either.
You've probably learned it more thoroughly than I have, or maybe we have different tastes. I find it just impossible to get into, it all rubs me the wrong way. It is so overcomplicated, and unlispy in the way that FORMAT and LOOP are unlispy, only much worse. I actually like FORMAT and LOOP but they're foreign implants amid the lispiness and that's how ASDF feels as well.
it's fukamachi-ware, guy has a bunch of own solutions in the web development space, i suspect driven by his specific needs. they are, fwiw, not as idiosyncratic as some of the artisanal common lisp one-man solutions can be, but they are distinctly and recognizably their own thing
I use his web stack in some of my projects. His stuff is pretty great, it's just that he basically doesn't write any documentation at all, and the small scraps of documentation he does write assume that you're intimately familiar with the details of his packages. So you need to figure a lot out on your own by digging through source code, which is very irritating when you're trying to be productive.
Those who still use Common Lisp or some variant of Lisp (other than Clojure), what's your reasoning behind choosing this language over the more mainstream options? I understand that Lisp has some powerful features, but almost everyone that I've talked to holds reservations for those.
For example, Lisp macros are amazing. BUT... this is not good for communicating code in big teams.
Or live (image) programs are an interesting concept, but git versioning them is much more difficult than traditional source code versioning.
Or hot code reloading, which sounds like a powerful concept except that companies would rather log and document their codebase. It also makes it challenging to keep track of the state.
If you force teams not to use macros, then a great reason behind s-exps goes away. And in general, if you take these features away from Lisp, I'm just not sure what use case Lisp can serve.
Lisp macros don’t actually make code harder to read and think about and, unlike the way widely used tools like Flink and Spark do code generation, it’s easy to see what the generated code looks like.
In practices, images are just a deployment tool in CL and most people don’t actually do “version control” by saving and restoring images or using hot code reloading. However, these features enable a development experience that I find much more pleasant than any other language I’ve tried.
I'm actually learning Common lisp for working on my side projects, and those are the reasons that really make the whole thing stick out and amazing.
* Live image and hot-patching is great. You edit it within the code, so if you're making a shooter game, and you're in a "draw HUD" function, you can just edit the code to be what you want and send that function to the running image, to update while you play. Any time you spot a bug, you can isolate, correct, and verify, while still running the game.
* Macros can be thought of as a way to make the code more readable. CL is almost thought of as a "programming language to make programming languages in, so you can solve the problem".
For instance, if you think about any amount of boilerplate. Macros allow you to write it once (because you have to) and then use it in different places at different times. CL makes boilerplate nonexistent.
I do this for wrapping up ncurses functions: to use colours, you must turn on and off certain bits and flags. You can imagine that there's complications when you want to draw a box, or colours text, or coloured underlined text, and there's all sorts of flags and state involved. To write it in C or Python would be a few multi line functions. In lisp it can be made as simple as (with-colours a-colour (draw-text) (with-underline (draw-text))).
> * Live image and hot-patching is great. You edit it within the code, so if you're making a shooter game, and you're in a "draw HUD" function, you can just edit the code to be what you want and send that function to the running image, to update while you play. Any time you spot a bug, you can isolate, correct, and verify, while still running the game.
I think this is an excellent selling point. Many frameworks (e.g. JavaScript frameworks) have file system watching implemented, but it is not the same.
I don't know what this means, how is this any different than any other function ? You document it, use it.. I think whoever has said that macros are a problem in teams is imagining problems.
> Or hot code reloading, which sounds like a
> powerful concept except that companies would
> rather log and document their codebase. It
also
> makes it challenging to keep track of the
> state.
You do the same you would for every language, you have test suites that exercise the code that you need and they need to pass. All code needs to pass on both the local system and CI, again a non issue.
> If you force teams not to use macros
This would be like forcing C++ programmers not to use objects, you could do it but you'd be doing it wrong.
There are many things to pick on common lisp for, but these are not them.
Start with CLOS being to too customizable, the loop macro being its own DSL, and the REPL/live image based development making you annoyed that other repls are considered equal.
The “downsides” you list are actually benefits that non–lisp users try to downplay. Especially macros. Macros give an individual the ability to write things in Lisp that would take a team in any other language. There’s nothing that says a team cannot gain the same multiplier.
The problem is that teams don't consist of the same people for eternity and when new people come in and you have to figure out the three DSLs that an old lisp wizard has cooked up you run into problems. With Lisp one person has the expressiveness of an entire team, but that also means it might take an entire team to figure out what one person did in the first place.
The success of Go stems from the fact that Go code in one place has a 99% chance of looking exactly like it does everywhere else. The power of lisp outsources complexity to readers, the simplicity of a constrained language imposes it on the writers, and because code is read more often than it's written, in a big project consistency beats expressiveness.
and you presumably have an argument for why that isn't true? I've seen exactly this at about three or four companies that had used Clojure. Successively switched to rewrite their codebases in Java because the Clojure codebase became too idiosyncratic as time went on and new workers had trouble being productive in it.
What is it with Lisp advocacy that turns people into religious advocates where market adoption seemingly doesn't matter, because it paints a very clear picture.
> what's your reasoning behind choosing this language over the more mainstream options?
Well, I could lie and list all of the great features of Lisp that others will have mentioned already, however the real reason I like Lisp is the S-expressions.
It may sound silly, but I really like the structure S-expressions give a program, which gels much better with the way my brain works than more 'conventional' syntax. Other Lispers tend to say that as you go on using Lisp the brackets disappear, but I like the brackets. They give me a sense of structure that I don't get from C, Pascal, Python, or Haskell. I dunno, maybe it's just my autism talking, but there you go.
It looks good to eyes that took a few days getting over their babby duck syndrome.
Has a useful numerical tower containing rational and bignums and an extensive math stdlib to act on them.
A powerful optimizing compiler (available to the user at runtime!) despite the deep dynamism. And pragmas (declarations) much better integrated with the language than in C/C++.
Gradual typing (missing recursive and user-facing parametric typing, I know...).
A stable standard with more than one quality implementation
A pretty good stdlib (missing some stuff due to its age, but well designed in what's there) with some hidden gems like the bitwise manipulation tools or compiler-macros.
CLOS/MOP is so flexible the sky's the limit (too flexible, some might say).
I could continue all day, especially if you want the bad and half-baked parts too, but my point is that there are other good points than the "esoteric" features.
Macros are fine and indeed part of the reason to use CL, just don't abuse them.
Common Lisp is the standard, it's ANSI. SBCL is a very robust, very performant runtime, with a pretty sweet price tag.
Sure, you don't want everyone to invent their own personal DSL:s for everything in production, but when you actually need macros the alternative is usually insane amounts of code that is rather inscrutable and quite hard to change across the code base.
And your developers can use the macro system to improve on their personal tooling, eliminating any concerns about it being detrimental to the team while still improving their working environment. You can also use the susceptibility to macros for static analysis and formatting tooling that makes the team's code more homogenous than it would otherwise be, like doped up linters.
You might not want to hot reload in production always, but when you actually need it, it could give a business edge or save the company from an impending catastrophe. It could also help improve working conditions in development and test, testing out patch ideas faster than subsequent builds would allow.
Macros enable communication, rather than hinder it. From the mundane ones defined as part of the language like defun and cond, to frequently helpful patterns like with-foo, to the more expressive ones like pattern matching or even domain-specific ones that can even let non-programmers contribute usefully.
No one versions their save-lisp-and-die blobs like source code, they're effectively binary releases. You could version them as releases.
Hot code reloading is not incompatible with logging, documentation, or normal file-based source control. You may have a misconception that we're just raw-dogging it with typing directly in the REPL window like you're more or less forced to with e.g. Python. Sure, sometimes, but the vast majority of the time no, edits are made to files in a text editor (and often saved) and sent to the REPL as complete forms. It's not challenging to keep track of state, either, especially when you can inspect everything whenever you want, trivially reload an entire system (what I often do after editing more than one file at once), and define more complicated state transforms ahead of time with things like update-instance-for-redefined-class. Rather, it helps avoid keeping track of so much state in your head, because when you're developing something and want to test something out, you just do it, you don't need to restart the application and rebuild the state to just what it was before the point where you wanted to test something different. Additionally, if a company isn't using some sort of hot reloading tool (like JRebel for Java), they're easily wasting 20% or more of developer time. (https://www.jrebel.com/sites/rebel/files/pdfs/rw-mythbusters...)
Besides that: I like dynamic typing, SBCL is default-fast unlike dog-slow Python and Ruby, optional types for speed boosts are there with compiler advice and ready disassembly views to help get and verify the speed (they also add a little bit of compile-time correctness checking, and options like Coalton are readily available for a more Haskell-like experience), the experience of conditions and restarts is way better than exceptions for normal development and debugging, debugging itself is more pleasant despite missing some tools because of the conditions framework and fully dynamic no-compromises hot reloading, code I wrote years ago continues to function, code that was written before I was born continues to function, Lisp's expression of OOP doesn't make me want to rip my hair out, and I rather like the uniformity of s-expressions over having to remember all the l-value/r-value/hidden constructor/operator/conversion/initializations/etc. details of other languages.
>Those who still use Common Lisp or some variant of Lisp (other than Clojure), what's your reasoning behind choosing this language over the more mainstream options?
One programmer can be amazingly productive due to how expressive and flexible the language is. I find my "developer velocity" (eyeroll) is much higher in lisp than in almost any other language.
>Or hot code reloading, which sounds like a powerful concept except that companies would rather log and document their codebase.
I think this is a benefit primarily realized during development cycles. You never "CTRL+C, edit, recompile, run". You just recompile the single function or statement and things just keep on trucking. If your production environment is containerized you'd obviously rarely be hot-reloading, but how often are you changing prod, anyway?
(just kidding)
Code at https://github.com/hraban/cl-nix-lite if someone is interested.
Obviously since this was written by Fukamachi you know it will be good. Much respect to the man.