Yeah, what I want is JSON with comments and nice multi-line string support. I don't like how much syntactic magic YAML does (I don't need or want the country code for Norway to be parsed as a boolean False value). I still don't know what the exclamation point does (e.g., !Ref).
And clearly YAML is the wrong tool for infra-as-code since CloudFormation has to build a macro system, conditionality, referencing, and a couple different systems for defining and calling functions (templates being one and their implicit functions being another). We also see tools like Troposphere and CDK which are effectively different ways to generate CloudFormation YAML via programming languages (or more precisely programming languages that were designed for humans).
And it's not just limitations inherent to CloudFormation--Helm has long had templates for generating YAML, but those also weren't sufficiently powerful/expressive/ergonomic so Helm3 is supporting Lua as well. And as I understand it, Terraform is constantly adding more and more powerful features into HCL.
So what's the solution? It's pretty simple--we should keep the YAML around, but it should be the intermediate representation (IR), not the human interface. The human interface should be something like a functional language[^1] (or an imperative language that is written in a functional style) that evaluates to that YAML IR layer. The IR is then passed to something like Kubernetes or Terraform or CloudFormation which understand it, but it's not the human interface.
As for the high-level language, something like [Starlark][0] would work well. It's purpose-built for being an evaluated configuration language. However, I would argue that a static type system (at least an optional static type system) is important--it's easy enough to imagine someone extending Starlark with type annotations and building a static type checker (which is much easier for Starlark since it's a subset of Python which is intended to be amenable to static analysis).
This, I think, is the proper direction for infrastructure-as-code tooling.
[^1]: Functional in that it is declarative instead of imperative--not necessarily that the syntax should be as hard to read as OCaml or Haskell. Also, while YAML is also declarative, it doesn't have a notion of evaluation or variables.
I do not understand the need for all of these different new language implementations and data formats. GuixSD vs NixOS already showed that Scheme is a superior solution as a configuration language, scripting language, template language, and intermediate representation. A single language that has 30+ years of successful production use, tons of books and documentation. Why re-invent the wheel in four different, incompatible ways?
Is NixOS a scheme? Anyway, we moved away from Nix because of all its problems (the language being only a medium-sized one). Also, I detest CMake, but by your own standard (longevity, popularity), it is better than Nix or Guix. Frankly those tools haven’t shown themselves to be “superior” in any meaningful way. Yes, they have been around for a while, but having been around for a long time and not enjoying any significant adoption is not very compelling.
That question does not make any sense. I am talking about Scheme the programming language: https://schemers.org/
NixOS is a GNU/Linux distribution built on top of the Nix package manager. The Nix package manager has its own custom configuration language. GuixSD is a GNU/Linux distribution built on top of the Guix package manager. GuixSD uses Guile Scheme as the package configuration language, system configuration language, scripting language, and implementation language for many of the system services (such as init). GuixSD does more things better with an existing standard programming language than NixOS does with its own custom programming language.
> Also, I detest CMake, but by your own standard (longevity, popularity), it is better than Nix or Guix.
What does CMake have to do with anything?
> Yes, they have been around for a while
What are you talking about? The first stable version of NixOS, 13.10, was released in 2013. GuixSD 1.0 was only released this May.
Yeah, I misunderstood your post—sorry about that. I thought you said “Guix and Nix show us that scheme is the answer”. In any case, I don’t know how you conclude that Guix won over Nix nor how you conclude that the winner is superior to all other configuration languages. It’s especially counter-intuitive that it should be an intermediate representation—the IR should not be executable, it should only be descriptive. The IaC technology shouldn’t need to interpret anything, and the language you pass into it therefore shouldn’t support evaluation/execution.
> I don’t know how you conclude that Guix won over Nix
There is no contest that I am aware of. I used GuixSD vs NixOS as an example of how adapting existing standards can provide a lot more benefits with a lot less effort than coming up with incompatible new languages.
> how you conclude that the winner is superior to all other configuration languages
Here is a condensed list:
1. 60+ years of successful use of S-expressions in all of the needed roles (programming language, intermediate representation, configuration language, template language, network protocols).
2. Many proven Scheme implementations available for use in any scenario, from clusters to microcontrollers.
3. Easily amenable to formal analysis and verification. Excellent tools such as ACL2 available and in use for many decades.
> It’s especially counter-intuitive that it should be an intermediate representation—the IR should not be executable, it should only be descriptive.
S-expressions do both.
> The IaC technology shouldn’t need to interpret anything, and the language you pass into it therefore shouldn’t support evaluation/execution.
That is an unexpected thing to say about an acronym that stands for "Infrastructure as Code." You cannot get automation out of static data.
I don’t find that list very compelling. Longevity in particular isn’t very interesting given that Scheme has gained very little ground in 60 years. That seems like an indicator that there is something wrong. I would sooner use Python or JavaScript, which are familiar to programmers in general and which have gained lots of traction in their respective lifetimes.
> S-expressions do both.
Right, that’s the problem. :)
> That is an unexpected thing to say about an acronym that stands for "Infrastructure as Code." You cannot get automation out of static data.
It’s a matter of architecture and separation of responsibilities. The IaC technology takes static data and automated the creation, mutation, and deletion of the resources specified therein. The client generates that flat data by evaluating a program over some inputs.
Yeah, I think Dhall is the right idea, but I think it’s going to have the same syntax/ergonomics issues that Haskell and OCaml suffer from. Our company invested heavily in Nix, but the developers really struggled with the expression language which seems quite similar to Dhall both syntactically and ergonomically. While Dhall might be great for Haskell/OCaml/F# shops, a configuration language isn’t the right place to push for new idioms or ways to think about programming.
We’ve been using starlark for kubernetes configs at my prev gig and I quite liked it. Open-sourced some of that stuff as https://github.com/cruise-automation/isopod (which is based on https://github.com/stripe/skycfg). I hear stripe are also using their thing with terraform although not sure to which extent.
TeamCity actually lets you set job configurations with kotlin in source control. The config is actually a kotlin dsl, so not only can you import settings, you can also have TeamCity generate projects, jobs, and settings from your code.
And clearly YAML is the wrong tool for infra-as-code since CloudFormation has to build a macro system, conditionality, referencing, and a couple different systems for defining and calling functions (templates being one and their implicit functions being another). We also see tools like Troposphere and CDK which are effectively different ways to generate CloudFormation YAML via programming languages (or more precisely programming languages that were designed for humans).
And it's not just limitations inherent to CloudFormation--Helm has long had templates for generating YAML, but those also weren't sufficiently powerful/expressive/ergonomic so Helm3 is supporting Lua as well. And as I understand it, Terraform is constantly adding more and more powerful features into HCL.
So what's the solution? It's pretty simple--we should keep the YAML around, but it should be the intermediate representation (IR), not the human interface. The human interface should be something like a functional language[^1] (or an imperative language that is written in a functional style) that evaluates to that YAML IR layer. The IR is then passed to something like Kubernetes or Terraform or CloudFormation which understand it, but it's not the human interface.
As for the high-level language, something like [Starlark][0] would work well. It's purpose-built for being an evaluated configuration language. However, I would argue that a static type system (at least an optional static type system) is important--it's easy enough to imagine someone extending Starlark with type annotations and building a static type checker (which is much easier for Starlark since it's a subset of Python which is intended to be amenable to static analysis).
This, I think, is the proper direction for infrastructure-as-code tooling.
[^1]: Functional in that it is declarative instead of imperative--not necessarily that the syntax should be as hard to read as OCaml or Haskell. Also, while YAML is also declarative, it doesn't have a notion of evaluation or variables.
[0]: https://docs.bazel.build/versions/master/skylark/language.ht...