> So for me, Common Lisp has actually shrunk without losing any functionality. This is not possible in languages without macros. Such languages are indeed doomed to either grow forever, or change in non-backwards-compatible ways (e.g. Python3). But with macros you can shrink a language as well as grow it.
Not true. If a language has adequate general constructs then you can replace language features with ordinary code written in the language. E.g. in Scala no-one uses "return" any more, because you can get the same functionality (in a better / more consistent way) using library types - just ordinary classes with ordinary methods, no macros needed.
With macros you've got a compiler support for anything you can imagine. And with macros you implement your DSLs as compilers, not interpreters, which is a huge advantage: you've got static verification, you've got any performance you like, and compilers are so much, much easier to implement than interpreters. So what's the point in doing the wrong, slow and error-prone thing instead of fast, easy and robust one?
If you're talking about this spire ( https://github.com/non/spire ) than it's using generics (i.e., poor man macros) and some real macros too, apparently.
> With macros you've got a compiler support for anything you can imagine.
Most of what you can imagine isn't useful or maintainable. Good tools should have more structure to guide you.
> And with macros you implement your DSLs as compilers, not interpreters
I'm not suggesting interpreters. If anything I'd say that macros - running arbitrary code at compile time - are more interpreterlike than what I'm describing.
> it's using generics (i.e., poor man macros)
Well if you're going to define every useful language feature as "macros" then of course you need macros to implement anything! But most of us consider generics to be different from macros.
> Good tools should have more structure to guide you.
It's not possible to have more solid and strict structure than with macro-based DSLs.
> I'm not suggesting interpreters.
You do. Either macros or interpreters. There is no other way.
> are more interpreterlike than what I'm describing.
You did not describe your approach to a problem yet. How would you implement an optimising BNF-based eDSL without macros? Spir and similar things are a totally different topic.
> But most of us consider generics to be different from macros.
Most people have absolutely no idea what metaprogramming is and how to implement DSLs properly. I would not refer to an opinion of a crowd.
> You did not describe your approach to a problem yet. How would you implement an optimising BNF-based eDSL without macros?
Let's get more concrete. What does the business requirement look like? Do you mean "a language that happens to be expressed in BNF", or are you asking for a DSL for expressing languages which itself looks like BNF?
> Let's imagine you want to embed parsers into your language. Choose any parsing algorithm you like, but parsers must be defined in a BNF-like syntax.
Doesn't sound hard to do in (macroless) scala. Just create objects and methods with appropriate names - and for the optimization part just ensure everything is lazy and preserves the structure so you have the AST available in the language and can do your optimizations at that level (which doesn't have to mean interpreting - we can use the type system to perform these computations at compile time[1]). The syntax will probably end up being slightly differently punctuated from actual BNF, which is a tradeoff for having syntax that follows the ordinary rules of the language and is accessible to e.g. an IDE.
I can agree that languages need to be able to perform complex transformations at compile time. But this doesn't have to be exactly the same kind as the compiler does itself, and as long as the language provides a sufficiently lightweight way of constructing an AST "in" the language, I think it's worthwhile making an explicit distinction between such ASTs and the AST of the language itself.
[1] I can imagine you objecting that this is just a macro system by another name, but it isn't (except in the trivial sense of turing equivalence). It has a different grain: it's more natural to create companion trees that mirror the structure of the AST exactly, and less natural to transform the AST by moving nodes around. And any such companions are explicitly distinct from the "original" tree, and the structure naturally lets you see both.
No, it's not easy, it's a barely usable hack. Coding anything on a type system level is like coding in Brainfuck or Unlambda, while with macros I can use whatever fancy DSLs I already have implemented for nice, declarative compiler construction.
Can you stop in the middle of your translation from BNF to low level code and dump a bunch of nice .dot files plus a tex documentation for the grammar? No. Your type system cannot do it, and you certainly do not want to do it in runtime, it's something to be done exclusively in compile time.
In a language with first-class functions, control constructs can be replaced by polymorphic methods - see smalltalk for the ultimate version of this. In scala the idiomatic way to do "short-circuit the rest of this in some cases" is usually for/yield with a type like scalaz's \/.
Not true. If a language has adequate general constructs then you can replace language features with ordinary code written in the language. E.g. in Scala no-one uses "return" any more, because you can get the same functionality (in a better / more consistent way) using library types - just ordinary classes with ordinary methods, no macros needed.