Complete Emacs-compatible Elisp implementation

    Thanks to the work of Robin Templeton, Guile's Elisp implementation is now fully Emacs-compatible,
     implementing all of Elisp's features and quirks in the same way as the editor we know and love.

https://lists.gnu.org/archive/html/guile-devel/2017-03/msg00...

lib-org-mode.

I love org-mode, but wish that some of the features could be ported elsewhere just to make the format more ubiquitous. Or a stand-along org-mode notebook server. Or support in other text editors. Or...

If I compare RMarkdown and org-mode, for example, org is strictly better for academic writing or anywhere else where internal cross-references are needed. Org isn't an astounding format, but it is quietly competent.

So it's stopped being an issue years ago!

You could force guile on everybody, just as e.g. perl5 did with their Test2 replacement, but not everybody will be happy with a 20% performance hit in the real world, and esp. when telling nobody about it. This would be pure marketing shit.

http://wingolog.org/archives/2017/03/15/guile-2-2-omg

Just because Guile implements Elisp the language doesn't mean there isn't a ton of work to be done on Emacs itself to swap out its native VM for Guile, and it seems nobody's keen on finishing up that work.

http://vanderburg.org/old_pages/Tcl/war/

Is wikipedia also doing a fud attack on Tcl (as it naturally could be the case), or is RMS post simply a less diplomatic and less friendly way to say that Tcl don't have the same purpose and use case as languages like c and c++? Any turning complete language can of course be used for anything, but looking at how something is used often give a good hint on where its strengths are.

What is not mentioned in that thread (at least not in RMS' original message) is that what triggered his edict was that somebody extended gdb with Tcl as the extension language. This was an absolutely reasonable use case for Tcl - the lightweight syntax makes it perfect as an interactive, embedded scripting language.

This discussion led to the creation of Guile, the withering of Tcl support in gdb, and eventually the introduction of a gdb that used Guile as its extension language — in 2014, 20 years after RMS decided that the Tcl extension (which I believe pretty much worked already) was unacceptable.

Of course, as astute readers will already have guessed, Tcl was BSD licensed...

Of course, as astute readers will already have guessed, Python is permissively licensed.

[0] https://github.com/8c6794b6/guile-tjit

[1]: https://www.gnu.org/software/lightning/

Having said that it seems to be in a POC state, but is very slow and needs optimization. These two wiki pages provide a decent overview: https://www.emacswiki.org/emacs/GuileEmacs & https://www.emacswiki.org/emacs/GuileEmacsTodo

This "Preview: portable dumper," thread from November 2016 and spin-offs appear to be the most up-to-date emacs-devel discussion on the subject: https://lists.gnu.org/archive/html/emacs-devel/2016-11/threa...

The older October 2015 "In support of guile-emacs" thread seems to have been written after development mostly tapered out, discusses some work to be done, and is a call for more development (which seems to have gone unanswered): https://lists.gnu.org/archive/html/emacs-devel/2015-10/threa...

All the work on the Emacs integration appears to have been done by Robin Templeton in early 2015 with no commit since May 2015: http://git.hcoop.net/?p=bpt/emacs.git;a=shortlog

Robin Templeton also seems to be the one which finished off the elisp compatibility on Guile here so it's probably because he saw that as important to get first.

Guile's strings (gnulib's) do not support that, and adding the support is quite a bit of works for all the corner-cases that emacs supports.

Now, emacs and guile do not necessarily have to use the same underlying string/buffer implementation, but it would definitely be a big plus, and I wouldn't be surprised the emacs maintainers would require it.

    Bonus track!  This release also contains a new experiment, a binary
    installation package for the x86_64 architecture.
    
    The GNU Guix project (https://guixsd.org/) has assembled a graph of
    package definitions (for example, GCC, glibc, Guile, and so on) and is
    able to build that graph in an entirely deterministic way starting
    from only a handful of trusted bootstrap binaries.  Guix recently
    added a "guix pack" facility that can export build products from a
    Guix system, including all run-time dependencies.
    
    We have used the new "guix pack" to generate an experimental binary
    distribution for the Guile 2.2.0 release.  If you are on an x86_64
    system running GNU/Linux, begin by running the following commands:
    
      wget https://ftp.gnu.org/gnu/guile/guile-2.2.0-pack-x86_64-linux-gnu.tar.lz
      wget 
    https://ftp.gnu.org/gnu/guile/guile-2.2.0-pack-x86_64-linux-gnu.tar.lz.sig
      gpg --verify guile-2.2.0-pack-x86_64-linux-gnu.tar.lz.sig
    
    If verification fails, then see above for instructions on how to
    import the appropriate GPG key.  For reference, the pack's sha256sum
    is:
    
      c707b9cf6f97ecca3a4e3e704e62b83f95f1aec28ed1535f5d0a1d36af07a015  
    guile-2.2.0-pack-x86_64-linux-gnu.tar.lz
    
    Then in your root directory -- yes! -- do:
    
      cd /
      sudo tar xvf path/to/guile-2.2.0-pack-x86_64-linux-gnu.tar.lz
    
    This tarball will extract some paths into /gnu/store and also add a
    /opt/guile-2.2.0 symlink.  To run Guile, just invoke:
    
      /opt/guile-2.2.0/bin/guile
    
    Voilà!

tl;dr: Because apparently some people believe that not only can they afford to exclude some users by avoiding zip's, but they must also bike-shed some more.

These days, some distros use lz and most use xz, probably mostly due to historical reasons.

1) Guile has no Global Interpreter Lock.

2) Guile is a scheme, so it is homo-iconic cf. https://en.wikipedia.org/wiki/Homoiconicity

3) Scheme (and lisp in general) are nice to write Domain Specific Languages.

4) Guile doesn't have a particular overhead for calling simple functions which makes it possibly as fast as C.

5) Guile has a very powerful object oriented programming framework beating by far Python and Ruby OO systems.

6) Guile is optimised for immutability which makes it for safer code.

7) Guile has Guix which has 'guix pack' command which is awesome cf. http://git.savannah.gnu.org/cgit/guix.git/tree/doc/guix.texi...

8) Guile has a lot of supporters cf. https://lists.gnu.org/mailman/listinfo/guile-user

9) Guile has awesome maintainers

Also, there is not a lot of packages. But programs written in Guile or using Guile tend to be of higher quality, cf. http://guildhall.hypermove.net/ and http://sph.mn/content/3e73

Honest question: How does that help nowadays? Certainly in the early 90s, browsing the GNU ftp site was terrific for discovery, and in the 80s, GNU tapes were even more powerful, but nowadays, GNU seems to be irrelevant for discovery. I can't think of a single software project in the last 20 years that I've discovered through its GNU association. Similarly, GNU used to be important as an infrastructure provider for open source software, but nowadays is one hosting option among many, and not a particularly attractive one.

GNU has never helped much in developing software, the individual projects always stood and fell with their maintainers. So it seems to me that nowadays people put their projects under the GNU umbrella mainly because they buy into a particular worldview of how software should be licensed.

So yes, being part of the GNU project is probably more of a cultural thing than an immediate practical advantage.

EDIT: Given that the original question appeared to imply "relevant as compared to other schemes"

Cf. is an abbreviation of "confer" which is Latin for compare and is used to introduce material that should be compared to assertions made previously.

Viz. is an abbreviation of "vidilcet" which is Latin for "namely" or "as follows" and is used to indicate a more complete statement or example of something that has just been asserted, whereas e.g. precedes examples that illustrate a point.

Viz. and cf. are often confused, as in this case.

If you have all these constraints:

- must be easy to use extensions languages functions from C, and the other way around

- must therefore be easy to convert values from the extension language to C, and the other way around

- must be possible to share this state in a multi thread program

- including, have several threads implemented in the extension language

- must be able to load code at runtime

Then guile is not only a good choice, it's sadly the only possible choice (or at least that was the case last time I checked few years ago). Many languages that claim to be good at extending C are lacking one of the above (usually the multithreading part; Lua, for instance, allows to start several VM in several threads, but they are isolated).

[0] https://www.gnu.org/software/guix/

What's that algebraic syntax'ed language he mentions?

Personally, I find Lilypond much better than Sibelius and Finale. In the past I have developed several Scheme snippets that allow me to apply complex layouts to my scores.

[1] http://lilypond.org/ [2] http://lilypond.org/doc/v2.18/Documentation/extending/index

I like Emacs and all, but I'm really not sure why we'd be excited about Guile. Andy is a really awesome guy but he's the only one contributing to Guile, so there's no way it's ever going to compete with things like Chez or Racket (which is moving to Chez soon as well).

Are you saying this is inaccurate? Andy is far and away the top contributor, it's not even close. It really does seem to be a one man show.

Guile 2.2 performance notes (FOSDEM 2016) https://www.youtube.com/watch?v=fU4Tly29Tps

- Performance: Guile 2.2.0 is halfways between the medium-speed Schemes and the high-performance Schemes. On par with Larceny, MIT, chicken and bigloo (and racket IIRC, but that’s missing in the benchmarks right now).

- Compatibility: Guile 2.2.0 is among the 10 most complete r7rs Schemes, missing only circular lists (the other failed task succeeds when allowing a longer task time).

However all this can be represented fully without parentheses — and this is possible with Guile using a reader extension without losing any of its power. This is realized, for example, in wisp — a language frontend for Guile: http://www.draketo.de/english/wisp

It’s a full Scheme, but without the parentheses (behind the scenes it simply infers the parentheses, adds them and hands the result to the regular Scheme frontend in Guile).

Wisp is also standardized as Scheme Request For Implementation 119: https://srfi.schemers.org/srfi-119/srfi-119.html

    display "Hello World!" ↦ (display "Hello World!")

I've heard this repeatedly over the years, but the explanation unfortunately always stops right there. Could you please give an example of why you'd want to change your source code programatically? It's always assumed that the reader implicitly knows why this is a good and important thing. Perhaps a practical example showing

(A) how this would work,

(B) what the benefit is, and

(C) how the added work in reasoning is worth that benefit.

Note: I personally like the parentheses, as they group everything together so simply.

This is my favorite example.

http://www.gigamonkeys.com/book/macros-standard-control-cons...

[Common Lisp's] DOLIST is similar to Perl's foreach or Python's for. Java added a similar kind of loop construct with the "enhanced" for loop in Java 1.5, as part of JSR-201.

Notice what a difference macros make. A Lisp programmer who notices a common pattern in their code can write a macro to give themselves a source-level abstraction of that pattern.

A Java programmer who notices the same pattern has to convince Sun that this particular abstraction is worth adding to the language. Then Sun has to publish a JSR and convene an industry-wide "expert group" to hash everything out. That process--according to Sun--takes an average of 18 months. After that, the compiler writers all have to go upgrade their compilers to support the new feature. And even once the Java programmer's favorite compiler supports the new version of Java, they probably still can't use the new feature until they're allowed to break source compatibility with older versions of Java.

So an annoyance that Common Lisp programmers can resolve for themselves within five minutes plagues Java programmers for years.

To create new syntax and be your own language designer. This is what macro systems allow for. Here's a very contrived and simple example. Guile comes with no equivalent of the `++` operator that we know from C, C++, etc. So in the event that we have some imperative code that is mutating a counter, we'd have to write something like this:

    (define counter 0)
    ;; do some stuff...
    (set! counter (+ counter 1))

    (define (++ n) (set! n (+ n 1)))
    (define counter 0)
    (++ counter)

    (define-syntax-rule (++ var) (set! var (+ var 1)))
    (define counter-a 0)
    (define counter-b 0)
    (++ counter-a)
    (++ counter-b)
    (++ counter-a)

The reason these syntactic abstractions are so easy to make is because of the homoiconic Lisp syntax. I hope this has made sense and is helpful. Moving on from this simple example, we can create entirely new languages that are embedded in Scheme if we wanted to, adding things that are too specific to a problem domain to ever be in the standard language's syntax but very useful for the problem we are trying to solve.

    (++ counter)

?

Lisp macros are something like C macros, chainsaws, hydrochloric acid, or anything else that's powerful-but-dangerous. This is to say that sometimes they are the one tool you have to use, but often they are unnecessary and should be avoided.

Coming up on ten years ago, I wrote this blog post on one of the dangers of Common Lisp style macros: http://www.mschaef.com/blog/tech/lisp/defmacro-coupling.html

Put succinctly, the problem I write about in the blog post is that macros are essentially always inlined into the output of the compiler. This has the effect of more tightly coupling the modules together than is evident in the surface syntax. (Note that macro invocation sites are syntactically indistinguishable from function invocation sites, which makes this problem worse.)

The upshot of this is what you'd expect: the extent of the logic encoded in macros should be minimized, with the macros translating the code pretty much straight away into something built on more functional abstractions.

This is not to say that Macros aren't useful... sometimes you _have_ to use them to achieve a goal. Just that they probably aren't as big a deal as might be expected given the amount of 'press' they get.

C macros have the issue that even when everyone involved in the creation and use of a C macro understands its pifalls, those pitfalls cannot be removed from the macro.

For instance, a certain C macro might evaluate some expression twice. Everyone knows that this is dangerous, but there isn't any way to fix it. They just document it.

ISO C itself says that getc may evaluate its argument multiple times; thus don't do things like getc(stream_array[i++]) unless you remove the macro definition wth #undef.

Lisp macros do not have issues that are unfixable in this way.

Sometimes they have issues that are difficult, though not impossible. Usually that occurs when, to be perfect, the macro would have to do a full-blown code walk. Macros are written that do code walks (for instance the iterate macro).

> Put succinctly, the problem I write about in the blog post is that macros are essentially always inlined into the output of the compiler. This has the effect of more tightly coupling the modules together than is evident in the surface syntax. (Note that macro invocation sites are syntactically indistinguishable from function invocation sites, which makes this problem worse.)

Before you apply macros, you need a well-designed (and documented, and versioned!) API against which the macros will write the code. If all you care about is what the macro syntax looks like and don't put any design into how the expansion works (beyond just massaging it so that it somehow works), then you may run into problems.

Macros don't introduce any problems that writing the same code by hand against the same API's wouldn't introduce.

If someone has to write the code, I don't see how you can get around it: it's either going to be a human, or a macro.

If your macros is 'fixed' to emulate function call semantics by evaluating its arguments only once, then maybe a function is a more appropriate abstraction in the first place. The whole point of macros is that they let you break the rules of function call application in hopefully useful and predictable ways.

Another way to look at it is that repeatedly evaluating an argument is what you do NOT want for a macro like 'getc', but probably what you DO want for a macro like 'repeat'. The danger lies in the fact that it's hard to tell the difference when looking at a call site in isolation.

Whether or not that danger is an acceptable risk is, of course, situation dependent.

All the programmer has to do is achieve run-time coverage to catch all the problems.

We could define a getc-like macro such that getc(*stream++) would diagnose, provided that the line is executed.

See sfx.h and sfx.c here: http://git.savannah.nongnu.org/cgit/kazlib.git/tree/

  ;; cond evaluated exactly once;
  ;; then or else at most once, not before cond.
  (if cond then else)

    (if* cond
        #'(lambda () then)
      #'(lambda () else))

   (defmacro (if cond then else)
      `(if* ,cond
          #'(lambda () ,then)
        #'(lambda () ,else)))

Works in C too, although not as nicely:

    void dscwritef_impl(const _TCHAR * format_str, ...);
    
    #define pdscwritef(flag, args) \
         do { if (DEBUG_FLAG(flag)) dscwritef_impl args; } while(0);

Huh? Are you saying the use of lambdas does NOT give if* the ability to control the execution of 'then' and 'else'?

My point is that if you're concerned about how often you evaluate a block of code (0, 1, or n times), there are ways to achieve this goal that do not require macros. (And consequently, the macros mainly serve as they should: to clean up the syntax, if necessary.)

I gave if as an example; it was not intended to be an example of a macro which has to go out of its way to ensure once-only evaluation.

There are common examples of macros that use machine-generated unique variables to hold the results of evaluating an argument form, in order to be able to insert that value into multiple places in the generated code. An implementation of with-slots likely has to, for instance.

In documenting a library of C macros, we cannot specify a strict evaluation order without seriously constraining which of those macros are actually implementable.

I think that came from me. My point was mainly that 1) control over evaluation is a significant reason to use macros 2) there are other ways to achieve that goal and 3) those other ways should be used to the extent possible. To me at least, this diminishes the value of one of the key headline features of the Lisp family of languages.

Note that this does not mean that I don't want to use the language.'I've maintained a personal and professional interest in the language that dates back over 25 years. When I have the choice, I usually reach for Lisp (really Clojure these days) as the most effective way to write the software I have the time and interest to write. It's just that the reasons for this don't center around the idea of compile time code transformation. (As useful as that can be when needed.)

> There are common examples of macros that use machine-generated unique variables to hold the results of evaluating an argument form, in order to be able to insert that value into multiple places in the generated code. An implementation of with-slots likely has to, for instance.

I do know this, because I've written at least a few of them myself.

https://github.com/mschaef/vcsh/blob/255afa60adf180e7e3e5b6e...

https://github.com/mschaef/vcsh/blob/255afa60adf180e7e3e5b6e...

Edit:

> ... insert that value into multiple places in the generated code.

Is that really what you meant to say? You're using a machine generated variable in a macro to 'insert a _value_ into multiple places in the generated code'? (As in, the value itself gets emitted in the generated code?)

The code I link to above does something slightly different. What it does is generate code that uses a machine generated variable to hold the result of a single execution of an expression. It then inserts references to that machine generated unique variable in multiple places in the generated code.

Lisp macros give meaning to syntax that doesn't previously have meaning. In this regard, they are the same as functions.

(foo (boonly) blarg) doesn't have any meaning because foo hasn't been defined.

We can fix that by writing a function foo. Then (boonly) and blarg have to be valid expressions and we are good.

Or we can make it mean something can by writing a macro foo. The macro foo is a function that will operate on the entire form (foo (boonly) blarg) and calculate a replacement for it. The Lisp form expander (a feature of the compiler or interpreter) will call the macro and accept its return value as the replacement for the macro call. Then, the replacement is scanned for more macros; the entire process removes all macros until all that is left is special operators and functions.

A practical example of a macro is the ANSI Common Lisp standard macro called loop which provides a syntax for iterating variables, stepping through collections, and taking various actions or gathering results:

  (loop for bit in '(0 3 7 13)
        for mask = (ash 1 bit)
        summing mask) -> 8329 

(Enter (Arne))

(Arne (Hello!) (Who are you?))

The above is actual code which makes Arne say two lines of text.

Essentially the Enter defines a new specialized control structure which is optimized for the task of defining lines of text for a character in an RPG to say.

This removes cognitive overhead while writing a scene: You only write what which character should say.

Essentially you invest into creating a better-fitting tool to simplify all following work. With Scheme you can take this further than with anything else — short of taking over the whole language implementation (which for example Facebook did with their new PHP implementations. With Guile you can take the path Facebook took, without first needing to be a multi-billion-dollar company). But as any other power, use this with care: You won’t want to make your code so much different from what other Schemers do that others have a hard time joining in.

¹: https://fosdem.org/2017/schedule/event/naturalscriptwritingg... — also see the video and the slides which show the difference between this syntax and examples from other methods, including one of my earlier tries with Python.

That can be utilized both at high level and low level. GOAL is a cool example: https://en.wikipedia.org/wiki/Game_Oriented_Assembly_Lisp. Basically you can express assembly code in LISP syntax and then you can create higher level constructs like if, while, for etc statements as macros composing these lower lever assembly instructions. Because data and code is the same in LISP that allowed Naughty Dog when they used GOAL on the Playstation 2, to swap chunks of code in and out of memory as needed easily to maintain larger levels than the competition. It is also something that allows you to change a running system. I believe there was a case of a malfunctioning satellite running LISP code, which was debugged and fixed while it was running. Shutting it down for fixing was not an option.

I would also add another benefit of only using parenthesis. It makes it really easy to make powerful tools for LISP. Have a look at s-expression based editing. Instead of working line by line, or word by word as we are used to with editors, these work with whole blocks of code at the time. Normal editor commands involve jumping word by word or line by line. with LISP editors you can view the code as a tree with keyboard commands for jumping between tree siblings, up to a parent or down to a child. Instead of selecting x number of lines or words, you can select a whole subtree and delete it, move it, duplicate it or whatever. I think they usually call it paredit. Here is an animated demonstration of how it works: http://danmidwood.com/content/2014/11/21/animated-paredit.ht...

Disclaimer: I never really got used to LISP myself. I think it is cool, but it was too big of a jump for me when I started from C++ background. However I feel more used to it now as I've programmed a lot in Julia which is LISP like but with more normal syntax. Also when I first checked out LISP I didn't understand the need to change the way you think about navigating and editing code. If you edit the way you edit normal code I think it easily gets confusing. You lose track of the parenthesis. I didn't know about stuff like paredit then.

Performance, macros are executed at compile time. Also, sometime the macro is easier to write that the equivalent non-macro code.

That's the whole syntax. So now you know how to program in LISP.

There's no statements in lisp, only the above expression repeated, which you can nest one in another, or sequence them one after another. They're called S-expressions.

For me, that simplicity is the first advantage. It's just very consistent and quick to learn. It also allows to embed everything together, like assign a value in an if condition.

The second big advantage, is that this syntax is very easily parsed into a list and can be easily changed programatically. Lisp lets you do that by defining macros.

A) It works by defining a function which takes the parsed code as an AST, and returns a modified AST. To transform the AST you have the full power of LISP availaible and some convenient operators that make it easier.

B) There's many benefits. Conceptually, it lets you change the evaluation order of code. Normally a function evaluates its arguments from left to right and then passes the resulting values to be evaluated by the function. 99% of the time this is what you want. But what if you want to short-circuit the evaluation of the arguments when one of them evaluates to say false. This would come in handy if writing the AND function. A macro allows you to do it. In other languages, you're given special operators for this kind of thing, but they're limited and you can not add new ones.

Another practical case is aspect oriented programming. Imagine you want to print the argument to all functions and all their return values. With a macro you can do this:

(print-steps (+ (- 100 23 price) (* 12 34 56 people))).

Print steps is able to inject a print statement at multiple code points, around each arguments and around every functions. Without a macro, you would have had to re-write all this manually, and then delete it again once you're done debugging. Some languages like Java give you a complicated framework which can do this to a certain extent, but aspect-J is a pre-processor, just like macros, it's just that the Java syntax makes this really complicated to implement, so you need a framework to do most of the work, or it's not practical.

C) That's up to you to decide. You don't ever have to write a single macro, and should only use them when they are needed and make sense. Now reading code with macros is trickier, because every macro could be a custom operator you have never seen before, or an aspect that does things you're not sure about. To me, this just becomes a thing of don't write crappy unreadable code. Which is true in all languages. I'd suggest teams standardize their macros, and document them well, so new members can quickly get familiar with your team's macros. Once familiar with a set of macros though, they make you a lot more productive and do make code more concise and even easier the read sometimes.

Btw, of modern Lisp dialects, I do not like Clojure for undermining this advantage by not syntactically grouping everything that is grouped semantically (e.g. binding pairs in let) just to use less parentheses. In my opinion, Scheme syntax (Guile is a Scheme implementation) is much better.

Javascript and C do it this way:

  if (foo) {
    bar(42, zonk());
  }

  (if foo
   (bar 42 (zonk)))

  if foo 
   bar 42 zonk

If it were to just be "foo bar 1", you couldn't tell the difference between (C style again for clarity): foo(bar(1)) and foo(bar, 1).

Some newer functional languages have a composition operator which allows you to write this without the brackets, but the brackets still are easier to follow in many cases.

Or `foo(bar)(1)`.

The syntax of the lambda itself in C++ is sort of funny: it requires to use all the bracket types at the same time!

  [](){}

IMO it's an open question. There is not much statically typed lisp AFAIK.

A good explanation is in Chapter 1 of "SICP":

https://mitpress.mit.edu/sicp/full-text/book/book-Z-H-10.htm...

Not strictly true... there is still parsing involved in reading Lisp data structures from a character stream. (It's just much less involved than in traditional infix languages.)

The way to think of it is this: 1) Lisp has a much more comprehensive (and read/write) syntax for its core data structures. 2) Lisp, the language, is defined in terms of those data structures rather than in terms of character sequences and grammar productions.

To see what I mean, consider this:

    (if condition 1 2 3)

It's the second-oldest high level programming language (after Fortran).

- Fortran: 1957

- Lisp: 1958

- ...

- C: 1972

The "family tree" analogy works imperfectly for programming languages since it is more of a directed cyclic graph than a tree structure – Fortran influenced Algol and then Algol in turn influenced Fortran.