Surprised that a thread about CSP in Lisp doesn't mention Clojure's core.async. CSP in Clojure is implemented as just another library and it's rock solid. While I don't claim to understand the implementation in detail, it's one of the more interesting and high leverage uses of macros I've seen in Clojure. It makes concurrent programming a breeze in Clojurescript as well, despite JS being single-threaded.
As someone who programs in both Clojure and Go there's absolutely nothing I miss from CSP in Go, and I would much rather do concurrent programming in Clojure.
As I understand it, there are multiple ways to do concurrency in Clojure. Is there a best way or a way that fits most use cases so that you should at least try feature X before features Y, Z, and/or roll your own?
There is no easy path you can follow sadly. It depends on your use case quite a bit. Clojure gives you a lot of options and you have to find your own way.
What Clojure gives you by default, are thread safe mutation constructs, atoms, agents and refs that help a lot if you want to do simply concurrent things without worrying about race conditions.
Clojure also has support to get some data parallelism, you can easily do a lot with reducers and transducers. Its quite nice.
For more complex stuff, core.async gives you a powerful CSP library, its as powerful as what the Go programming language gives you (I think its nicer because Go has statements instead of expressions).
Because Clojure is on the JVM, you get access to all the Java stuff, java.util.conncurrent, and you can get a lot of powerful tools there (ThreadPools and stuff like that).
The reason, sadly; is that I lack experience with clojure.async, I lost my motivation before I got there; a couple of years ago it was still too experimental for my taste, didn't offer enough advantages over CL, lacked fundamentals like error handling and was tied to the mountain of complexity called JVM.
To clarify, I meant this thread on HN when I said thread. I don't find it strange at all that that your article didn't mention it. Thank you for writing it!
"One of the things that Go (and Erlang) got right was using channels as the main mechanism for synchronising and communicating between threads."
Erlang doesn't have threads and doesn't use channels either.
Erlang has lightweight processes, which might map onto OS threads (but user code has no control over this).
Erlang uses async communication which is always non-blocking (of course, you can emulate sync communication, the gen_server:call behaviour in the OTP library does that, for example).
Only Erlang got it right, not Go. And channels were not even invented for real life programming, they were just an easy notation for mathematicians. But an interesting thing about concurrency is that the Erlang myth is actually true, getting deeper into concurrency always leads to reinventing Erlang. The road typically goes from synchronous multithreaded applications to asynchronous single threaded to event loops to higher-order abstractions around events to wrapping them into killable process/context abstractions to adding message passing and to reinvented Erlang.
This illustrates the power & the problem of Common Lisp: the power is that it really is easy to add channels (and pretty much every other feature) to the language yourself; the problem is that it's generally easier to write your own code to do this than to use (or fix …) someone else's.
I've experienced exactly this: I too looked at ChanL. In my case it had a bug for one of my use cases, and it was honestly easier to roll my own channels (with, of course, their own bugs) than to understand & fix ChanL's bug.
The problem is that this leads to a proliferation of different libraries, all doing similar things. The power is that one really can write just about anything, and it will work well enough for your use case.
As you sort of imply, it doesn't help that each implementation is half-baked, incomplete, and unmaintained after 16 months.
I've seen similar libraries that do the following, for example:
* Serialization and deserialization
* Promises
* Generators
* CPS
Somehow none of these "100 line wonders" make it into idiomatic usage, despite their purported utility.
These channels as presented don't solve many problems well. They will be very slow and contentious, they will be expensive to make, and they will not guard against memory ownership issues because you're just passing pointers to objects around, which message passing is supposed to solve.
But, it'll become another library on Quicklisp that 3 people use and that people's applications will inadvertently depend on.
Lisp is a fantastic language for many things, including a plethora of production applications, but it being a local optimum for exploratory programming means we get lots of very incomplete, scratch-an-itch code that is pawned off as a library intended for mass consumption. In fact, the author links to his GitHub repository, where you find exactly this random assortment of disparate utilities.
If I had my way, the rhetoric of this post would be closer to a pedagogical exposition of channels, how they're used, what their benefits are, a proof-of-concept implementation, and a non-trivial example application. I would vehemently stay away from having the thesis be anything about Lisp or the ease of implementing small POCs in Lisp.
it doesn't help that each implementation is half-baked, incomplete, and unmaintained after 16 months
I couldn't agree more, that's the real problem with CL and the reason why I keep using scheme and Racket. (Well, that and the antiquated package/unit systems with multiple options and caveats.) Just take a look at cross-platform user interface library bindings, there are seemingly plenty of options, but in the end they are all some sort of unmaintained hacks that may or may not work on some implementation/platform combo. Threading itself is another example, by the way. Almost any addition to the outdated standard is a mess that sometimes works, sometimes doesn't work.
It's a pity, since CL implementations are of such a high quality (e.g. SBCL) and it offers everything a programmer can dream of.
This implementation is yours to maintain, that's why it's simple enough to make that possible. There is plenty of room for simple. They allow building networks of cooperating threads, which is the purpose. Erlang isolates, Go doesn't, in the end I like the choice.
Copy-pastable blog post code is the antithesis of good software engineering practices. DRY, encapsulation, reusability, broadly applicable abstractions, and so on should ideally be heralded as the thing to strive for.
I don't agree. As long as you understand the code and it solves your problem, owning is an advantage. Why is it so important to you that no one steps out of line and tries to use their brain as more than an answering machine?
> As long as you understand the code and it solves your problem, owning is an advantage
Because the cost of acquiring an understanding of code that solves your problem might be much higher in Lisp than other languages, if the hypothesis is correct that the language facilitates the creation of bad half-baked libraries.
Just to give a somewhat figurative example, the jungle is a much more hospital environment for life in general than temperate environments (it's warmer, full of nutrients, bathed in constant solar energy).
However most humans didn't manage to really expand into the jungles successfully until the advent of modern medicine, because the life there was so prolific that there were all sorts of nasty parasites and diseases, and everything you build sinks into the jungle within years.
Lisp is like that jungle -- everything just grows fantastically easily, but paradoxically that makes it harder to separate the chaff from the wheat.
But these are symptoms; you're still basically arguing that Lisp gives the user too much power. And that's a valid perspective. I don't agree that a language can give the user too much power. Building tools for someone who's supposedly not as smart as your self is a great ego boost, and that's about it.
The less I have to maintain fundamental abstractions, the more energy I can spend building on top of these abstractions. There's a name for this: NIH [0].
While there are some folks [1] that have the time, energy, and brilliance to build things from the ground up all themselves, I unfortunately do not. Moreover, in a collegial work environment, this means the burden is on me to test, document, communicate, and educate on this library. There's enormous additional complexity and risk that comes with building from scratch things like these which change the whole programming paradigm of your application—again, especially in a coworking environment.
It is a fallacy that you are not maintaining something with your dependencies. It is the hope that you get nothing but benefits from depending on someone else to build strong dependencies. It is as true of a reality that they are pursuing goals that are not yours.
It is odd, because the "micro library" world ostensibly fixes this by greatly limiting the scope of a dependency. However, it also encourages chaining yourself to many other entities. And it is always the mistakes that people remember, such that anyone that has been burned will remember how it was enabled by micro libraries.
You are maintaining the API boundary between your application and your dependencies. I certainly do not maintain (in the most common sense of the word) my operating system source code, my compiler source code, my server source code, etc.
I have no problem with micro-libraries. In fact, I developed on the notion of the micro-est of libraries: a library generator that gets down to function-level dependencies [0].
Exactly. And I wasn't trying to be anti micro services or libraries. Just pointing out that confirmation bias is a big reason some folks are against them.
It is amplified when folks push them with no caveats. Some of us remember being burned in ways this allowed.
What you're saying is really that it's too powerful for it's own good. And I agree. But what's good for Lisp isn't necessarily good for me, I prefer more power to one size fits all libraries. I think it comes down to experience in the end; the more you have, the more unwilling you are to give it up.
I disagree about it being "too powerful for its own good". If it was, we would have a magnificently efficient and useful library come out of this that brings Go- or Erlang-style programming to Lisp. But alas, this code does not.
A lot of hardcore Lisp aficionados do the equivalent of a mathematician writing a "sketch of a proof" and saying "left as an exercise", without ever writing the details of the proof. It's occasionally aggravating when you pull in a library and discover that this is the case.
To be clear, there are many Lisp folks do not do this. Some go the full mile and implement something completely and robustly. Edi Weitz has been the canonical example in the community.
Completely, what is that? Doesn't that depend on context (as in problem being solved)? Simple is often faster, and these are pretty fast without even trying. There is plenty of room for simple, fast enough code. Owning code you understand is an advantage.
Completeness means being as performant as possible on all the possible axes.
You might want simplicity, but users also want performance. So a complete solution will be performant, but simple.
You have one use case, but other users have others. A complete solution will work in as many use cases as possible given its constraints.
Scala's parser combinators are a good example of a complete solution. It offers the "elegant" solution of parser combinators. But it also offers an implementation of this using things like Packrat parsing that are extremely efficient.
The non-complete example of this is someone who writes a parser combinator library that is simple, but simply not performant for real world use. For example, you might write a simple version in Python, but not properly apply TCO and so your parser can't handle deeply nested structures because of a stack overflow.
---
With regards to owning code, I remember hearing that it took 9 years of the initial publication of quicksort for there to be a bugfree implementation of it. Even easy things can be tricky.
What's comprehensive? Doesn't that too depend on context? I'm with Kent Beck on tests. I'll test as much as I have to too move forward with confidence. In the end, my goal is to write working code, not tests.
> A lot of hardcore Lisp aficionados do the equivalent of a mathematician writing a "sketch of a proof" and saying "left as an exercise", without ever writing the details of the proof. It's occasionally aggravating when you pull in a library and discover that this is the case.
That's exactly "too powerful for its own good". It's so easy to write a half-assed sketch of a library that nevertheless gives you some new powerful feature, that some people end there, and publish the sketch that was enough for their use case.
I guess I should have mentioned that Lisp is my language of choice. I like the ease with which I can implement anything. But it does have a cost: everyone else implements everything himself, too.
I actually think Quicklisp has helped. It's even easier to do (ql:quickload "foo-lib") than it is to write my own foo library.
Everybody can do that stuff in Clojure too, and it does not suffer from the same thing CL does. It seems to me that it is a culture/ecosystem problem more then anything else.
> The problem is that this leads to a proliferation of different libraries, all doing similar things.
This is not "left-pad" either. I'd say you have the same problems in C or C++, where people are reluctant to depend on libraries they don't control or appear to be bloated (boost, etc.). As far as Lisp is concerned, this post talks about the problem more directly: http://fare.livejournal.com/169346.html
In this case particular case, what is being shown is that channels in itself are pretty simple concept, especially when done on top of language with preemptive multithreading. Same thing can be implemented in essentially any imperative language with necessary primitives by code that looks more or less exactly same.
> The proposed implementations are unusual in that they use a very simple purely functional subset of the well-known programming language LISP. This will afford additional excitement to those who have access to a LISP implementation on which to exercise and demonstrate their designs.
I think a lot of people in this thread are being absolutely ridiculous. It's not easy to implement channels in Lisp because of Lisp, it's because the work is already done for you.
What mainstream language doesn't have structs, condition variables, locks, threads and... Wait, I think that was all of it? Oh, and linked lists. I mean come on, you could do this just as easily in Java.
And all this talk of the Lisp curse, Jesus Christ. I mean, there's a point in that there are a lot of lackluster libs (but what language doesn't have that), but Quicklisp and a switch in Lisp culture means that there's a large amount of collaboration (and usage of libs) between programmers. There's the potential to start to use the same libraries and a will in the community to do so.
Really interesting read for me as an experienced Go developer currently learning Lisp. Not sure if I will ever use Lisp or this for a real project, I see it more as some kind of Sudoku in code, and a indirect training ground for Elixir (which is a wonderful complement to Go for backend tasks).
- Use QuickLisp as much as possible. In many cases you don't need to add your own lib. Read Quickdos [QDC].
- Comment your code well, even if others don't ever see it lest you could not be able to understand your own code later. What's the purpose of the function, which arguments are expected, of which expected type are they, what are the possible results, possible side effects, etc.
- Read and understand how Lisp's package system really works. This is important to avoid strange behaviors in your own code.
- Don't use Hyperspec too much because it is really confusing. Use proper reference manuals like Peter Seibel's Practical Common Lisp [PCL] and Edi Weitz' Common Lisp recipes [CLR]
- If you cannot get used to Emacs (which is a wonderful tool with a steep learning curve, really the best editor ever) then you can use proprietary IDE's like LispWorks or Franz Allegro.
Do it. I spent years nervously fingering On Lisp on lunch breaks without really getting anywhere. But Paul Grahams early essays kept pulling me back into Emacs, and on each iteration something fell into place. It me took a long time to feel comfortable, but I wouldn't trade the feeling for anything. It's an alien chain saw; chain saws are intimidating enough in themselves, one with completely alien controls and instructions even more so. To paraphrase Graham, power looks weird from below.
I once tried ChanL but it was buggy, and concurrency bugs are the worst to debug.
lparallel on the other hand was solid to me, though I didn't like its API and had to (trivially) build my own message-passing abstractions on top of it.
As someone who programs in both Clojure and Go there's absolutely nothing I miss from CSP in Go, and I would much rather do concurrent programming in Clojure.
Rationale: http://clojure.com/blog/2013/06/28/clojure-core-async-channe...
Code: https://github.com/clojure/core.async/blob/master/examples/w...
Presentations: https://github.com/clojure/core.async#presentations