This comes up on HN periodically. Here are few points to keep in mind:
- In 1994, when Andy started on GOOL, the tools around C/C++ were far more limited. Basically all we had was GCC; there wasn't even a decent debugger in the beginning if I recall correctly. So it was less crazy to make a whole DSL for game control code then than it may seem now.
- There was no GC; the PS1 only had 2MB of RAM so we had to be incredibly careful and precise with memory management. It was also slow (33Mhz) so GC pauses were obviously not going to be OK.
- At least in Crash Bandicoot (I didn't work on Jak), lisp was not used for the rendering engine, the collision detection, or the camera control code -- that was all written in C (or MIPS assembly). Lisp was used for character control logic and various non performance intensive tasks, like the load/save interface.
- It really wasn't that hard to "make a whole new language" in our case because Andy used Allegro as the basis for the compiler -- and of course with lisp, code is data. :)
It was a labor saving device and a boon to rapid prototyping of character behaviors to have a simple very high-level language for this. This advantage was ultimately offset over time as the platforms got faster and gained more RAM. But at the time, there was no Python, Lua, etc. -- and even if there were, they'd be too RAM-intensive to use. (Maybe embedded variants of these languages would have worked, though.)
Please add 2012 in the title. Also this was previously submitted without comments.
The topic was old... but I am glad someone commented on the downsides of implementing your own language/tooling and what happens next to the team; for example:
> Also, it took over a year to develop the compiler, during which time the other programmers had to make do with missing features, odd quirks, and numerous bugs.
That makes me thing only ultra-flexible teams or huge companies fit the bill into the approach of creating such a technical debt as a compiler can be.
Hell, I was reading the source code to moonscript (compiles to lua) the other day, and you'd be shocked how intelligent some of the components are! Even for such a simple little language.
I also think flexibility may not be the best word, but I know like what you're saying. But it's more like cooperation, I think. I once wrote an exploit (for work, legit, no funny stuff) for basically a cnc machine with a php cpanel ui built on to it. Me and two other guys at the time wrote this big kit for it, and the kicker is we had to do it remotely because we had one car between the three of us and we were perpetually broke at that time in our lives. I think the car even got like broken into or something once. Anyways, we were using a diagnostic computer that could read serial data from the machine, ssh-ing into that computer, and iirc there was a buffer overread that we were using to jump the memory out, and just sending each "batch" as a numbered tarball. Thank God that that little computer existed, because that was the ticket for making the whole thing worked. The place was out in the boonies of Charlotte, and public transit wasn't what it is now. The factory only ran two shifts, so this entire setup turns off every night, so we needed to make efficient use of the diagnostic time.
It was a mess, but we were all on the same page to reach the goal: hack that damn machine, get paid and get a better gig! We communicated, we helped one another. It was confusing, we had a spiral notebook that we passed around (too lazy to do data entry, too broke to go get a photocopier) with different opcodes and offsets for the cnc.
Talk about technical debt, that exploit was the rattiest, grimiest, duct-tapiest, ugliest code I've ever shipped in my life. We got patted on the backs for that job! I remember we set up some garden variety packet filter or something for the php they had and it was a breeze. Getting that exploit working to get them bought in... What a nightmare.
Ofc, teams are one of those things. Ymmv. It takes the right people at the right time to make those miracles happen. You just can't recruit that kind of providence, and you can't train (imo) that type of battle-buddy synergistic workmanship. Sometimes it has to be you and your friends to make it work.
It does sound like the programmers were still productive in that time, though. The beneficial features of the custom compiler came iteratively throughout.
I remember running into GOAL and GOOL a ways back and was fascinated with it. One thing I never got good info on was how they handled their memory. Generally game developers frown upon garbage collection, so I'd love to find out if GOOL/GOAL did anything to target that specifically. I tried reaching out to one of the developers on his blog about it but never got a response
If I recall correctly, they used manual memory management for many things. But regardless of specific tooling, you can still do memory pooling in GC languages just as well as in manual ones.
The Lispiness that they took advantage of was metaprogramming via s-expressions, an interactive REPL, and dynamic updating of functions during runtime (which would require some form of GC in function memory footprint, but wouldn't be a drag on normal gameplay timing).
From what I could glean from the GamaSutra link, memory management was one of GOOL/GOAL's weak points:
GOAL's ability both to execute code at the listener and to replace existing code in the game at run time introduced the problem of memory usage, and more specifically, garbage collection. As new code was compiled, older code (and other memory used by the compiler) was orphaned, eventually causing the PC to run low on free memory. A slow garbage collection process would automatically occur when available memory became sufficiently low, and the compiler would be unresponsive until the process had completed, sometimes taking as long as 15 minutes.
Wow, I remember reading that postmortem when it first came out.
I was still in university at the time and at the time I was only marginally aware that Lisp existed as an alternative to the Java/C++/Pascal we've been taught.
After reading the postmortem I became very interested in this mysterious Lisp language that was being used by the experts.
A. Using Lisp with a couple of minor convenience macros can be a very good idea.
Z. Writing an entire language and ecosystem just for a game is insane.
These are not binary choices: there is a very smooth continuum between these two, and is one of the major advantages of Lisp. Naughty Dog just took too many steps from A towards Z. Each step probably seemed reasonable at the time...
> Writing an entire language and ecosystem just for a game is insane
Don't think so. It was actually a series of games and critics generally like the game engine aspects. Naugthy Dog developed for new platforms and their games were written to show off the capabilities of the platforms. Since there were no off-the-shelf game development systems like that - where one could develop inside a running game on prototype hardware - they developed their own. At that time there was a lot more Lisp education and they had their computer science education from MIT.
Looks like the GOAL/GOOL runtimes and dev environments worked very well, created well received games and they sold a lot of them. Naughty Dog used Allegro Common Lisp on the development environment side and their own Scheme-based runtime on the game hardware.
Later when they were bought by Sony, they switched to C++ to be able to share code inside Sony - which then seemed to be problematic and they returned to Scheme for some stuff:
Infocom, Sierra, and LucasArts all built proprietary VMs and it seems to have worked out very well for them. People are still making new games that run on Infocom's Z-Machine, and ScummVM has helped keep those LucasArts games running to the modern day.
Unreal Engine was very successful with a custom scripting language up until UE4, too. And Naughty Dog hardly seems to have suffered from their decision to build a Lisp-based system -- apparently they're still using Racket to script their (wildly successful) games.
Really, it's only quite recently that the off-the-shelf tooling like UE4 or Unity has gotten sophisticated and powerful enough that people can make complex games without having to build a little software ecosystem around them.
- In 1994, when Andy started on GOOL, the tools around C/C++ were far more limited. Basically all we had was GCC; there wasn't even a decent debugger in the beginning if I recall correctly. So it was less crazy to make a whole DSL for game control code then than it may seem now.
- There was no GC; the PS1 only had 2MB of RAM so we had to be incredibly careful and precise with memory management. It was also slow (33Mhz) so GC pauses were obviously not going to be OK.
- At least in Crash Bandicoot (I didn't work on Jak), lisp was not used for the rendering engine, the collision detection, or the camera control code -- that was all written in C (or MIPS assembly). Lisp was used for character control logic and various non performance intensive tasks, like the load/save interface.
- It really wasn't that hard to "make a whole new language" in our case because Andy used Allegro as the basis for the compiler -- and of course with lisp, code is data. :)
It was a labor saving device and a boon to rapid prototyping of character behaviors to have a simple very high-level language for this. This advantage was ultimately offset over time as the platforms got faster and gained more RAM. But at the time, there was no Python, Lua, etc. -- and even if there were, they'd be too RAM-intensive to use. (Maybe embedded variants of these languages would have worked, though.)