A few strong opinions, a lot stronger than they were in my 2017 talk https://youtu.be/_gmMJwjg0Pk, surely that must be the experience speaking :)
- Creating a DSL is a commitment. It is never done, it's either in development or abandoned. Plan for this, both in terms of team size and knowledge management.
- Don't allow escape hatches or ways to access arbitrary functionality in the host language. When (not if) these are used, they ruin backwards compatibility and make maintenance of DSL code a nightmare.
- Don't use a PEG (parsing expression grammar), they are terrible for backwards compatibility. Hand-written parsers are even worse. Using an LR parser generator is worth the time investment.
- Support static analysis, or at the very least don't make it impossible to implement later. It improves productivity, and every error message is an opportunity to teach the DSL to the users.
- You need a language server. The bare minimum is: highlighting, auto-completion, hints on hover, find references, rename.
- It should be trivial to reproduce a production run on a development machine. This means the language should be deterministic, and the production context should be preserved for later reuse, and the development machine should be safe for accessing production data.
- Thousands of unit tests, thousands of regression tests.
It's fine to break these rules, if you can live with the consequences.
And here's a brand new one from this year: use RAG to have ChatGPT "read" your documentation, and ask it to produce scripts for you. When the script doesn't compile, feed the error message back to ChatGPT and try again until you get a working script. Think deeply about why ChatGPT made some mistakes, as it's often the case that humans would have the same misunderstandings. Improve your documentation, error messages or syntax to eliminate them, and then try again to see if it works. We have been doing this for 6 months now, and there have already been some significant improvements that are only obvious in hindsight. And we've now opened a PhD position to investigate whether we can have a separate "LLM-friendly" syntax for the language, dedicated to making code generation easier.
> Hand-written parsers are even worse. Using an LR parser generator is worth the time investment.
I'm curious, what makes you say this? From what I've read online secondhand, people tend to recommend handwritten recursive descent parsing and Pratt/TDOP parsing, because it's easier to add good error messages and later add context sensitive productions (e.g. the way Clang resolves class-wide declarations while parsing C++: https://eli.thegreenplace.net/2012/07/05/how-clang-handles-t...).
He explains it in the linked video presentation which is worth watching: the value being that your parser generator will complain about any ambiguity in the grammar (shift-reduce conflicts) rather than having that show up as a “wat?” at runtime.
You might argue that this argument reduces to checking the grammar using static analysis similar to the order generator. But then handwriting the parser is still extra work and risk.
What I've seen most nontrivial projects do (and have done in a commercial product myself) is to start with a parser generator then move to hand written parsers when UX and polish become a bigger priority than exploring the problem space.
The parser generator grammar spec can then be used to generate random test cases and compare the output of the new parser with the old one to make sure they're identical.
- Any expression that may likely include a user-generated variable should have a strategy for how to inject it safely. To avoid sql injection class of attacks. Native templating is the best. If string templating is the only way out, the rules for escaping should be clearly defined and ideally functions for doing so provided by the reference implementation.
Not like the geniuses at Atlassian who came up with JQL that refuse to document how it works, instead delegates all security to the user model and “don’t run queries with any data that you didn’t provide yourself”.
Is an LR better than a parser/combinator.
Seems like there are pluses to rolling your own parser/combinator over the time to integrate with a yacc or other library.
I am a bit biased (in the sense that I wrote my own SLR parser generator twice, in C# and F#!), but the benefit of LR is that you can know whether your CFG contains conflicts, which is a game changer for preserving backwards compatibility when you change the syntax later during the life of the DSL.
But if a parser combinator library may support converting the resulting combined parser into an eBNF grammar, and check whether that grammar contains conflicts.
Hum... A grammar conflict on parser combinators always require that you step down from the parser abstraction and resolve it by hand with basic language functionality, doesn't it? It's something quite hard to notice.
Are you concerned with the documentation getting out of sync with the actual language?
In a situation where I'm adding a new feature to the language, with a new syntax that causes me to add a new rule to the grammar, I'm concerned that the new rule will accidentally "capture" some code that already exists in the wild, that was previously derived by another part of the grammar.
To give a very ugly example, if you have a language with function calls f(expr, expr, expr) and you want to add tuple syntax to expressions with a brand new rule:
expr := expr COMMA expr
Then you might have accidentally turned all functions into unary functions, as the tuple rule captures the "expr, expr, expr" part and leaves you with f(expr).
Good lord yes on the language server. Reduce the friction involved with writing code.
I'm frustrated with terraform-ls right now. It doesn't work with private module registries, reloading each module in the .terraform dir over and over. Which impacts performance by maxing out a CPU core.
And yet a feature of Terraform Cloud (TFC) is that you can use private registries.
Writing Terraform is already plagued by friction. Adding TFC adds to the friction. The language server then not loading private modules adds even more.
Hopefully OpenTofu helps in that regard, but I don't think they've got anyone working on the language server either.
> but I don't think they've got anyone working on the language server either.
Not yet, but it's on our radar. We're focusing on the stable release + registry for now (eta mid-December), but we're planning to work on the language server eventually.
If you have any more frustrations with the language server, feel free to respond here, and it'll help us find avenues for improvements.
It's really not a frustration, I should have mentioned that you were going to look at it once things settled down :/
I'm incredibly supportive of your efforts, I just wish I was a good enough coder to pick up golang and make the language server better instead of just griping about it.
I created a ticket about dedupe, but I have to admit it's a very low priority compared to so much other stuff.
I think my only other suggestion would be to concentrate on making the tf development cycle quick and easy. The easier it is for people to write the code, the more code they'll write, even with copilot or other helpers.
I'm in the midst of creating a DSL (it's my first try) and this sounds like really good advice. I'm not clear on the third item though - by LR parser generator are you talking about something like ANTLR?
Not having ambiguities is actually the main selling point of PEGs. If you have two rules A and B that can both match the input, then a CFG A|B has an ambiguity (two possible derivations), but a PEG A/B explicitly says that the A derivation is chosen. The good part is that unlike a CFG, the PEG doesn't require you to go and fix anything (the / operator already did that for you). This makes the initial implementation of the grammar easier.
On the other hand, if you already have code in the wild that uses the old grammar G1, and in order to add new features, you introduce a new grammar G2 that is a superset of G1. You need to know if any of the existing code has a derivation in G2 that is different from its derivation in G1 (as that would cause backwards incompatibility). With a PEG, there's no way to tell, so you have to check this by hand (and mistakes are easy). With a CFG, you know that backwards incompatibility happens if and only if G2 has conflicts, and those conflicts are precisely the cases that are not backwards compatible.
A PEG is not actually a generative grammar but a domain-specific language for specifying top-down parsers. So they are free of conflicts and ambiguities by definition of their semantics.
To be perfectly honest, I'm not sure what that means. I'm new to language design. I can describe what I'm trying to do.
I'm creating a UI description language for designers. The goal is to let them use terminology and mental models familiar to them to describe their designs in code, instead of a visual tool like Figma.
A trivial example would look something like this:
component Button {
elements
shape btn-container
text btn-label
style btn-container
fill-color: blue
style btn-label
content: Click Me
text-color: white
}
The output of the language would be an intermediate representation, I'm imagining a JSON object or something with a very specific schema. This can then be transpiled into any format the developer wants - you could build a transpilation target for React, Vue, Svelte, plain old html/css, etc etc.
So I'm in a weird spot where I know what I want to make, but I don't know any of the conventions or tools common in the language design world, because I'm just stepping into it.
Have you ever designed a library? If not, then my recommendation would be to start with it, for example in Typescript if you are familiar with it. I.e. build a set of functions that allow you to create the datastructure (AST) above. That is the most important part, encoding it into json or something like that is very easy afterwards.
If you feel comfortable with your result, then you can think about turning it into a DSL. Reasons to do so include mostly syntactical concerns (since you don't want designers to learn typescript, even if it's a subset). You will have to do a lot of the things that the host-language otherwise does for you though, including syntax-highlighting, type-checking and so on.
The host language is the language that you are using to write the compiler/interpreter. I suppose the grandparent is asking the question in order to recommend a parser generator for your use case. From your GitHub profile, I assume it's JS or TypeScript ?
Hey Victor, these are amazing tips. I am currently building a DSL for LLMs, and would love your feedback/consulting. I could not find your email online.
Please feel free to reply to this or reach me here:
@eating_entropy on X
joshcho@stanford.edu
Would love advisement from someone who has thought about DSLs extensively!
Unfortunately, they don't mention the most easy way of defining a DSL: by doing it in the host language itself. Not all programming languages are capable of allowing that, but some do, and in that case it is by far the most easy and safe solution.
It is still much much easier than combining existing language like adding OCL into another language, as suggest in the paper; and you get compiler/IDE support, tooling support and everything else from the "host language" out of the box.
I'd go a step further, and suggest that most languages are able to embed some level of DSL, although you might not be able to reach quite the expressiveness that you had in mind.
Generally a combination of block closures and being able to chain methods together is usually enough to create something reasonably expressive, although even then there are alternatives - I've seen DSLs in Python that use the class syntax and metaclasses to define quite complex this. You won't be generating your own syntax or anything, but you can often get pretty far.
This is definitely the best idea for "drop down" DSLs (by drop down, I mean that you are working in a general-purpose language and then, for one specific situation, you drop down into the DSL because it is more productive).
I think the article is more oriented towards standalone DSLs (where the entry point is the DSL itself), as those get actual maintenance benefits from not being able to invoke arbitrary code in the host language.
Yes. My point is just that I would judge that in 90% of the cases, "drop down" DSLs are the best solution for the problem at hand. So when talking about DSLs they should be mentioned, or at least the term DSL should be defined accordingly, which did not happen in the paper.
Doesn’t necessarily have to be the host language. Just any other existing language.
If it’s static and declarative, just use yaml. (Yes it sucks, but you get it point)
If it’s imperative, just use the host language.
Then you have some hybrid solutions. HCL allowing rich expression in an otherwise very declarative language. A big list of constant dataclasses in python is another quick way to define complex data structures without having to reinvent anything. This even gives you type safety for free.
Heck, MongoDB managed to build big business on a database whose only query language is super verbose json. This isn’t just negative, because by leveraging existing language, a lot of tooling around is ready to plug in on day one.
The _by far_ easiest way is to use the host language itself.
Using _any_ other language, e.g. yaml, forces you to do a lot of things that you get for free when using the host language, including syntax, type-checking/linting, build-system if you need one and especially all the tooling that comes with it for free (e.g. autocomplete in IDEs).
In fact, there is almost no benefit at all of using yaml _for DSLs_.
(And yes, there is widely-available syntaxhighlighting for yaml and others, but that doesn't over anything specific for the DSL, or otherwise you will have to customize it)
Host language is rarely ever declarative. Declarative languages are often necessary because they don’t leave any escape hatches that tempt you into writing incorrect code. Build system dependency graphs one common example. It’s also needed to pass data between different systems without recompiling and to constrain the usage boundaries, especially around security.
Imagine how absurd it would be if mongodb queries were written in the host language C++ instead of json.
For in-house stuff agree. There is no point to make excessive config layers if only developers will touch it anyway.
> For in-house stuff agree. There is no point to make excessive config layers if only developers will touch it anyway.
Yeah, I'm specifically talking about in-house DSLs or those that are only used by/from specific users. That doesn't have to be developers, but they should be using the host language as well - and in that case, an imperative host language can still make it able to use a declarative DSL. (in the most basic way for instance: generate a datastructure that essentially describes a json/yaml or whatever)
For non-inhouse DSL, I agree that it is usually worthwhile to make them standalone.
Alan Kay describes OOP as defining interactions with/between values via mini languages/algebras. In a language like Smalltalk (using whitespace and mixfix syntax) this very much feels like crafting a lightweight/hosted DSL. Other languages require a lot more squinting, to get past the host language's punctuation and ceremony.
For example, compare something like the following:
canvas drawA: circle at: topLeft in: red.
canvas.draw(circle, at=topLeft, in=red);
I think the discoverability of Smalltalk code within the IDE lends nicely to that too. It'd be quite obscure to many these days, but there's something to be said about the ability to browse and experiment with a catalogue of classes like a graphical REPL on steroids.
Raku has a built in Grammar capability that integrates well with regex / tokens / rules in an composable OO context. As mentioned you get all the IDE support (IntelliJ based) including a grammar inspector tool.
I first saw this talk of DSLs with Ruby, where it is/was popular to use missing method and others to build your own "language".
I do not mean it in a negative way, but at the time it seemed like calling a utility class DSL was excessive. It just seemed like regular programming, using the language constructs. It also was not enforcing many boundaries (there are escape hatches).
Looking at all of this, would it not be easier to encode the DSL in something like json, and turn it into a data/format problem? Something like terraform. The implementation can then be in any language that can read json. One could imagine, for example, regex defined in json as a DSL.
I think every language (DSL or not) should support translation to/from a simple data representation for its syntax trees (whether s-expressions, JSON, XML, etc.). Not necessarily the same representation, but just something structured that existing tools can walk and transform, such that (a) we don't resort to grepping through bytes (and manually filtering out matches that are comments or strings) and (b) adding a keyword to a language won't break all existing tooling that's unable to parse it.
I would say JSON + schema, rather than just JSON. Like S-expressions, JSON saves you from having to implement a tokenizer, and a tokenizer is one of the easier parts of a DSL. You still need to implement something like a syntax, because things that are handled by the syntax now get pushed into later stages of the DSL. However, if you can provide a JSON schema for your language, then the problem is mostly solved !
On the other hand, it's not enough to read JSON, you need a JSON parser library that provides you with the position (line:column) of every JSON value, so that you can emit user-friendly errors.
I've created a few DSL's and I've regretted not closely emulating an existing language that users already know.
When creating a new DSL you should find the language that is most familiar to your users. That could be Python, Javascript, or SQL. There is a good chance though that it is spreadsheet formulas.
Whatever that language, align capabilities you add to your DSL with the syntax and function naming of that language only diverging where needed to add the custom capabilities which motivated the DSL.
The closer you align with what your users already know, the more easily they will be able to adopt it. An added benefit of this that emerged in the past year is GPT's will be able to write your DSL with less prompting.
I built a DSL for a Visual Novel game I am making, by using ANTLR for parsing and then executing it using C#.
Are there generic frameworks for implementing static analysis that exists out there ?
I did a rudimentary version of it with some simple checks, but I have troubles wrapping my head about checking conditional branches and stuff like this.
I want to build a DSL for a really specific reporting use case. It's daunting though. Not sure if what I have in mind is a true DSL, perhaps just a souped up builder pattern.
Perhaps, it's a build your own report thingy for end users or perhaps internal (non technical) support to use. I've never seen these things work so I'm deferring this feature until the end as best I can.
I wonder if those users could learn SQL+Jinja2+a few spreadsheet-like helper functions to use in the templates?
End users can use Excel, which is close to a real language, but declarative and reactive and heavily framework-ed.
I'm not sure how a DSL would be much easier than Jinja2. I'm thinking of adding user Jinja templates to my SQLite browser utility.
Zero code GUI only config is my favorite way to do things, but after that I'm not sure why you'd want to not just jump straight to a mainstream language that people can Google.
In the 2000s Myspace users learned HTML just fine. Mainstream languages are already designed to be easy for anyone to use.
- Creating a DSL is a commitment. It is never done, it's either in development or abandoned. Plan for this, both in terms of team size and knowledge management.
- Don't allow escape hatches or ways to access arbitrary functionality in the host language. When (not if) these are used, they ruin backwards compatibility and make maintenance of DSL code a nightmare.
- Don't use a PEG (parsing expression grammar), they are terrible for backwards compatibility. Hand-written parsers are even worse. Using an LR parser generator is worth the time investment.
- Support static analysis, or at the very least don't make it impossible to implement later. It improves productivity, and every error message is an opportunity to teach the DSL to the users.
- You need a language server. The bare minimum is: highlighting, auto-completion, hints on hover, find references, rename.
- It should be trivial to reproduce a production run on a development machine. This means the language should be deterministic, and the production context should be preserved for later reuse, and the development machine should be safe for accessing production data.
- Thousands of unit tests, thousands of regression tests.
It's fine to break these rules, if you can live with the consequences.
And here's a brand new one from this year: use RAG to have ChatGPT "read" your documentation, and ask it to produce scripts for you. When the script doesn't compile, feed the error message back to ChatGPT and try again until you get a working script. Think deeply about why ChatGPT made some mistakes, as it's often the case that humans would have the same misunderstandings. Improve your documentation, error messages or syntax to eliminate them, and then try again to see if it works. We have been doing this for 6 months now, and there have already been some significant improvements that are only obvious in hindsight. And we've now opened a PhD position to investigate whether we can have a separate "LLM-friendly" syntax for the language, dedicated to making code generation easier.