I don't think pasting tokens with ## is such a common practice at all, and it certainly must be used prudently in order to avoid producing unreadable code. Myself I used it for one little thing (I posted it above) to get nicer code. I needed it because I had pointers of various types and wanted to associate data of a fixed type with each of the pointers. So I simply made an enum of BUFFER_pointername symbols which I then can use to associate arbitrary data with them. I cannot go the other way (i.e. making a link from the extra data to the pointer) because then the pointer type information gets lost - I would need to go with a void pointer and add the actual type in lots of places.

I also don't like putting the pointer together with the extra data because that means dependencies, and also I want a simple bare pointer to the array data (instead of a struct containing such pointer) that I can index in a simple way.

I also don't like the stretchy_buffer approach where you put the metadata in front of the actual buffer data. Again, because of dependencies.

The alternative would have been to go for C++ and make a complicated templated class. I don't use C++ and templates are a rathole on their own. So a single ## in my code solves this issue. I'm happy with it for now.

You're comparing against a different thing.

I'm suggesting that in lisp or rust macro land, your macro is a lisp or rust function. So in sane c macro land, your macro is a c function.

    macro macrofun(AST* node) {
        node->name = strcat("BUFFER_", node->name);
    }

Literally just use a subset of c syntax on a c ast represented as a tree of node structs. It need not be blazing fast, it runs at compile time.

C++ constexpr is close, although decidedly less powerful, but is still a huge win over c macros.

You keep making excuses for why you're doing these things, and I don't really care why you're doing them. I'm saying you shouldn't need to, and that the interface with which you would solve them should be less awful and inherently error prone.

But making a less error prone interface takes more up front complexity. You've elsewhere claimed that this has positive externalities ('it forces you to understand c sytax better'), but I'd reverse that and claim that

1. It violates a common api ("languages are expression oriented"), and is therefore both astonishing and leaky

2. It doesn't force you to understand the language better, it prevents you from being productive until you understand a set of arbitrary rules (you list these elsewhere) that aren't necessary for normal use. Token based macros require you, the user, to understand how c is parsed, AST macros don't because they do it for you.

No need to get all worked up. As I said, I'm not idiomatically against AST manipulation at all. Just that I don't have a big issue with C preprocessor, which is a good solution for the simple cases. It doesn't improve the situation if you replace ## by programmatic access -- it's a little more involved but on the upside you can maybe expect slightly better error messages in case of wrong usage.

In the end, there are zero problems with my approach here, either. And the CPP doesn't encourage you to get too fancy with metaprogramming. Metaprogramming is a problem in its own because it's hard to debug. I've heard more than one horror story about unintelligible LISP macros...

Note that I am going to experiment with direct access to the internal compiler data structures for my own experimental programming language as well. But you need to realize that this approach has an awful lot more complexity. You need to offer a stable API which is a serious dependency. You need to offer a different API for each little use case (instead of just a token stream processor). If you're serious like Rust you also need to make sure that the user doesn't go all crazy using the API. Finally, it's simply not true that you need to understand less about parsing (the syntax) with an AST macro approach. The AST is all about the syntax, after all.