I think the problem with claiming this is the "next big paradigm" is that the average programmer, and frankly, even the merely above-average programmer, is not going to be competent to design a non-trivial special-purpose language of any value.
Once you step out of the domain of totally-trivial languages, language design immediately becomes tricky, subtle, and prone to exotic interactions and quirky tradeoffs that even our absolute best teams of language designers can only mitigate, not eliminate... and it's not all going to be "absolute best" teams, after all.
And... then you want multiple languages to be sitting there interacting, too? That's not even possible without somehow limiting those interactions ("thy languages shalt have lexical binding that works thusly"), and now you're just making another meta-language like Lisp, which, presumably, doesn't fit the bill. Either you cut off the diversity of the language or you get the sort of evil interactions the likes of which have never been seen in a real language. The only middle ground there is to do both at once.
If your paradigm requires a genius to use it, that's not a fair comparison against other paradigms; start a genius out on OO and they can get pretty far there, too.
Here's an article from 2003 that I think is still very relevant inspiration for what's to come. I think we're "getting there" and I plan to be part of this revolution ;)
I especially liked page 3. It's hitting nail after nail right on the head.
edit: On second thought I'm not sure we're getting there because I'm not aware of any IDE that comes close to approaching what I have in mind (which is pretty close to what the article is describing). I guess I just have those concepts in mind so much that I think everyone else sees what I'm seeing. Anyway.
I thought pg was thinking this way with arc. I remember reading something where he was planning to add some kind of syntax definition capability to arc, but haven't been able to find the article since.
It sounds like all he needs is a VM, such as the JVM.
For example, the JVM gives us:
Java, Clojure, Jython, JRuby, etc.
All of these languages are compatible with each other and can easily extend each other. Especially with Clojure, it would be trivial to implement another DSL on top of these, but the platform has always been there.
If you have ever used JetBrain's IntelliJ IDEA, you will understand why standard text editors are grossly inefficient. IntelliJ is a Java IDE, but imagine for a moment that it was an "all JVM powered languages" IDE. It's a very interesting vision, and I for one am confident that JetBrains can produce something compelling. Their products are top notch.
I'm trying to think of an easy way this idea can be hacked together with existing tools, instead of writing a bunch of new stuff.
Could this be handled by macro programming on a lisp machine? Lisp is essentially the parse tree, which all languages have, so it provides the syntax agnostic representation. Macros allow you to easily specify DSLs, plus the macros themselves are lists and thus manipulable by macros. The lisp machine is essentially emacs that really is an OS, so the entire programming environment is programmable. But, emacs on top of some shell isn't so bad either.
The only thing missing is the syntax layer. Maybe regexps could suffice? Also something like LaTeX can provide a rich variety of symbols and symbolic structures for syntax.
Regexps alone are a really poor fit for dealing with recursive structures, such as finding matching pairs of braces. You can use them for a lexing step, but you're probably better off using something like yacc or LPEG for the parsing.
Forth, Factor, and other stack-based languages also have this same property of "syntax agnostic representation". While on the surface they're kind of a backwards Lisp (due to postfix notation, with intermediate results pushed on a stack), both are dealing with nested lists of symbols, and very adept at dealing with things expressed the same way.
That's true about the nested structures. I was wondering if it is worthwhile to make lexing and parsing first class and changeable at runtime, and whether regexps would be a good way of doing this. The problem with relying on an external program is that making syntax conversion first class gets much more convoluted.
But, it isn't clear to me whether it is a good idea to make everything about LOP first class. I do like that feature of Lisp though.
Forth is not adept at dealing with nested lists of anything. It doesn't have linked lists, garbage collection, or traditionally even heap allocation. It does have recursion, but it doesn't traditionally have local variables, and mutual recursion (really handy when you're doing anything related to ASTs) requires using DEFER or something. Do you have a lot of experience writing Forth?
Forth words are nested lists of other words (before compilation). It's not as good at dealing with nested data structures as Lisps, agreed, but the poster above is asking about syntax and DSLs.
Before compilation, Forth words don't exist. During compilation, they are flat lists of other words, not nested lists of other words. After compilation, they can be just about anything. Forth is great for embedded DSLs but it's not because of its great ability to deal with nested lists of words.
Or, possibly, perl6 rules. One of the things I keep meaning to have a go at is taking Larry's gimme5 stuff (a slow but working implementation of much of the rules system in perl5 he's using to bootstrap the grammar) and bolting it back into perl5 so we can experiment with it more easily.
I'm pretty knowledgeable about Lisp so if these ideas could be implemented just by using some existing tools together I'd probably know. This article goes much further than that.
As high-level as Lisp is, it's still a general-purpose, turing-complete language and as such there's always going to be a DSL that's better suited than it to solve a particular domain-specific problem.
While it's true that you can embed DSL's in Lisp easily with macros, that doesn't address the editing issue at all. Current IDE's are not so easily extensible to support the high-level semantics of new DSL's, so even if your DSL is 20x more expressive than vanilla Lisp in your domain, you lose a lot of that productivity to the lack of tool support for it which we've grown accustomed to having with general-purpose languages (syntax highlighting, dependencies, reverse-dependencies, parameter hinting, refactoring, etc).
Do you know if anyone has tried abstracting out the elements of a good IDE? Your list looks like it could be implemented with a table structure. All languages have similar dependency structures (as far as I know), so the IDE would just need to know the right syntax elements. I'm probably way over simplifying though. I'm not broadly experienced in many languages.
I'm not sure if you're joking. It has always been the case that you could run Lisp on stock hardware. Lisp machines were just an optimization for back in the day when hardware was slow and Lisp compilation techniques weren't well-known.
Wrong. The lack of tool support for DSLs is a real issue. We've barely scratched the surface of what IDEs can do. We often have to deal with lackluster editing support (especially so in the case of DSLs) and I find this deplorable.
As an example, if a DSL's function C references a lisp function F, the IDE should integrate the two languages enough that when you ask who calls F the IDE will list C among the calling functions.
edit: Suddenly I have a feeling people will have to see the power of a "real" IDE for themselves before they see that all the seemingly small improvements are really worthwhile. It's a bit like explaining Lisp macros to a beginner: the simple examples are easy enough to understand but aren't very compelling, while the complex, more worthwhile examples are too difficult to understand. So let's talk again in a few years when I have an implementation ;P
Once you step out of the domain of totally-trivial languages, language design immediately becomes tricky, subtle, and prone to exotic interactions and quirky tradeoffs that even our absolute best teams of language designers can only mitigate, not eliminate... and it's not all going to be "absolute best" teams, after all.
And... then you want multiple languages to be sitting there interacting, too? That's not even possible without somehow limiting those interactions ("thy languages shalt have lexical binding that works thusly"), and now you're just making another meta-language like Lisp, which, presumably, doesn't fit the bill. Either you cut off the diversity of the language or you get the sort of evil interactions the likes of which have never been seen in a real language. The only middle ground there is to do both at once.
If your paradigm requires a genius to use it, that's not a fair comparison against other paradigms; start a genius out on OO and they can get pretty far there, too.