The backend of Skyrim (and other Bethesda games) is largely a database. Bethesda calls them .esp files, but they're relational in a way. There has been a large open undertaking to produce tooling for this, such as xedit (Pascal) and z-edit (JS).
When you open a container in the game, and it doesn't have a specified set of contents, the contents get loaded by picking randomly from a category linked to in the database. If, in one of the db tools, you follow the link from the container you can see what objects could load in it and by what likelihood. You can then follow those links through the db over to the object itself, which is also made up of db entries.
From the scripting system you can access the db for some amount of functionality. Their db doesn't contain the models or textures or strings. But it does contain the location of every tree in the game. If you wanted a mod with a dozen golden crates to go find, you would use the db system for the logic.
However it is not 100% documented what kind of optimization this "query engine" has implemented, there are some vibes that sometimes it could be faster than a full scan.
Remember: game devs don’t create intelligent game entities; instead they create the illusion of intelligence. Game dev is a big smoke an mirrors charade.
Any system that could be queried needs to be built and maintained. Creating game entities that have models of feeling or have opinions about the groups of people they belong to has to be modeled and built. Unfortunately game dev is very informationally and financially sparse so creating these systems is a huge undertaking.
I heard a joke once about OOP: if you want to create a form that accepts payment information you must first create the universe. The joke speaks to a danger of overdoing your conceptual modeling. The real trick of a true craftsperson is to create a model that is excellent at solving the problem on hand.
What I’m saying is we’re a long way away from game entities behaving like west world hosts and embracing the ECS paradigm isn’t the answer. Just like how switching from OOP to functional doesn’t actually solve creating a profitable business.
The researchers behind the project have since moved on, but the idea of a db underpinning a game engine came up over a decade ago in Walker White's work at Cornell. Unsurprisingly, there's a whole lot of ways in which expertise from the db community can be commoditized to allow game devs to focus on the game, rather than hand optimizing code. For example, query optimizers can make it easier to scale up complex agent behaviors [0]. The project also spawned some interesting work on checkpointing, distributed replication, and computation over uncertain state.
There's a solid game design framework underlying that choice, beyond just being smoke and mirrors.
Most single player games are not "true games". They are puzzles. This is also how players perceive them.
As defined by Chris Crawford a game requires competition. Multiple agents who play, which creates the possibility of win and loss.
In most single player games, the game agents are not equal competitors who can win. They are only obstacles and challenges for the player. Even if the game is built with difficulty in mind.
So the mistake of thinking of actually intelligent game agents is like thinking why doesn't the Rubik cube ever win. It's not why someone plays it.
Notable exceptions are where software is used to substitute a player in a multiplayer game, where they do usually maintain the game structure and compete.
Maybe I focused too much on the terms. They certainly don't think of the word puzzle, but they do treat them as that.
Someone playing Zelda doesn't go into it ever expecting Ganondor to try to win. The player either solves the game, or they stop playing. But they can't lose, which is why I called it a puzzle.
In comparison to something like Chess AI where the computer agent is actually trying to win and make you lose.
So you'd find players asking whether it's single or multiplayer and competitive or casual and so on and they're really trying to place the game in that categorization. Is this game just a series of challenges for me, or are there opponent agents trying to win.
I put that in quotes because I didn't want to emphasize the terminology but maybe it didn't work. Whatever they call them, players treat games with equal opponents who can win, and games that are only a series of challenge entirely differently. The issue I'm talking about is developers trying to insert competitive agents into games which gamers expect to be a series of challenges.
Thinking back to the example in the article, there's no reason why the game designer can't just script in a player ambush, since it's a plausible thing to have happen next. I don't think you need any kind of "game intelligence" to tell a compelling story.
It depends on what kind of game you want. In many games scripted events are fine, but a game like Skyrim would feel much more alive if NPCs could do more than walk back and forth all day. In those cases scripting becomes a lot of work very quickly - the Mass Effect series had a lot of different outcomes based on player choices, in the end the final game was disappointingly linear and predictable and handling thousands of possibilities turned out to be a bit much. Immersive game worlds could use a lot of improvements.
The problem is that this is hard. Just think about how game engines consist of a graphics engine and a physics engine. You would need various other engines that can just be plugged in. A realistic economy requires a complicated supply chain for example. This means just so your NPC vendors can sell a sword, you would have natural resources that can be extracted, manufacturing processes that require inputs and machinery/tools and labor and all of this for something that the player clicks "buy" on and never looks back.
This problem has been solved many times, for example the goal-oriented action planning method allows dynamic behaviour and real games have implemented this.
But you are correct in that this means more effort for developers and doesn't sell more lootboxes.
In 2013 I wrote about "game interfaces for work" where work interfaces should act like games. Real time strategy games make you feel empowered, if you could queue up real work in a units runqueue. Of course you'll have actions besides "build" and "attack" to map to the richness of the world.
I've played around with using SQLite to store the scene graph for a game and using SQL to query/manipulate it. Very elegant and interesting ideas start to emerge, but performance quickly becomes a monster problem even if you run it in-memory. For simpler scenes, it definitely works though.
The reason I went down this path - I was thinking about streaming gaming and how we could potentially keep 1 representation of everything in memory and service hundreds or thousands of players with the same data set. Think tables like Scenes, Entities, Viewports, etc. Big picture, I felt like you might be able to pull off something at scale with a bunch of read replicas over a big boy SQL database.
This kind of approach maybe good for very big companies, but it absolutely kills small companies and indie devs.
Indie Devs should be thinking of games as products to complete first and foremost. Don't get bogged down in articles like this, or what's the best tech.
Just ship it.
"Make games that are fun to make." Sometimes that means "thinking of games as databases."
100s of games released every day across Steam and the App Store. Who cares? I guess they shipped, but then what?
> Don't get bogged down in articles like this, or what's the best tech.
Jonathan Blow is excited about Jai. That makes him wake up every day and commit. Secularly Jai will not make the game better. But intellectual stimulation helps you ship.
So your advice is a negative sign in front of the right answer: it's pithy and simple, but it's 200% wrong. It's so far off the mark and so steeped into the exact kind of hustlebro culture that is responsible for the 100:1 ratio of bad games to good.
> ...maybe good for very big companies
Big companies struggle to innovate. They worry way too much about shipping, deadlines, whatever. So they wind up with the least risky and conservative game design. They are interested in deep tech because, if you're making the 100th CS:GO clone, it makes the work intellectually stimulating enough to make you wake up every day and grind 65h/wk for Riot's garbage pay, not just because it makes some kind of secular sense.
> Since he started working on Jai he hardly shipped any game.
That's an opinion. Maybe if you looked closer, you'd see: if it weren't intellectually interesting, he wouldn't ship any games, as opposed to a bunch of really cool and fun ones.
I can only speak personally. As a game designer, I have never regretted "not" "shipping." Spending the time on design instead, I feel my opinions are far more valued by my design and directing customers, who have bigger budgets and thus will have automatically greater success marketing, polishing, "shipping," etc. anyway.
If you think you need to ship in order to find out if something is fun... well, there's your problem. What do you think game design is, an A/B test?
In contrast, my former colleagues who emphasized shipping, in the last decade, they have spent 100-10,000x the budgets on putting kind of dull stuff into the App Store. And it was only marginally more stuff.
Anyway, I wouldn't ever ask them for an opinion about game design, or whether a prototype is going to be fun, but I would ask Jonathan Blow, so there's that.
A remaster of an old game in 7 years. So yeah, "hardly shipped any game".
> why do you perceive this as a problem?
It's probably not a problem for him. It's not a problem for you either, if your studio can afford spending 7 years on a tool that doesn't generate cash flow.
Again I have nothing against Blow. But it's a very good example about how much time it takes to make your own tool chain, even for someone who's smart and experienced. For most of us, we'd better utilize boring techs to ship lowkey fun games.
AAA studios have shipped a lot of games with broken online services, gameplay, matchmaking, etc. No AAA studio has ever shipped with a broken cash shop. Its all about priorities.
Well, I mean, software in general is one of the only items I can think of where you can intentionally sell someone something that is broken and promise to fix it in the future if they keep paying you, otherwise make it unusable for them.
Sure, the mission should always be: shipping fun profitable products. Doesn't hurt have solid code architecture from the get go though. From what I understand ECS are very flexible and probably your best bet. It's often hard to see where a game is headed, which can turn thoughtless code very ugly very fast. Still, many games get shipped like that, but if you're in for the long run you'll eventually have to maintain it. So why not consider the advice from industry veterans and read about a few proven design patterns.
Absolutely, however game developer forums, discords, etc are full of people who obsess over the "ideal" way to do things rather than ship their game and go bankrupt or destroy their lives doing it.
So it's extremely common polite etiquette in the indie game developer to remind people that the end goal is a fun product that you can sell. Many many people lose a ton of money during indie game dev, and it's always a tragedy.
If you want to use a no code tool like playmaker over coding, then do it.
As long as you ship the code. Just keep shipping out your game.
Haxe has CastleDB [0], which I've used once, and it's great (it only powers your game assets statically, and isn't written to during gameplay, or used by the running game itself (I think)). (On Godot now, so mostly using Resources, which can be structured data on steroids.)
I didn't know about CastleDB. The fact it uses JSON to store the DB data and schema itself is a nice feature. I've been using Sqlite for a similar purpose to store structured data and load configuration, but the biggest downside is having configuration in a .db file that's a binary blob doesn't play nice with Git or other source control.
I wonder how well SQLite would just handle this whole thing.
Seriously.
Compile a bunch of prepared statements at load time to take the parsing and execution planning out of it, keep it all in memory... On the surface it seems like it would work.
And of course, saving the entire game state would be as simple as just flushing the DB to disk.
One of the reasons for using an ECS is cache coherence. All your game state is lumped together in one chunk or memory. Lookup times are near instant as a result.
Moving to SQLite makes theoretical sense, but you lose the benefit of cache coherence as a result.
I'm not familiar with the guts of SQLite enough to say, but wouldn't this be true for an in-memory database too, if you structured the "tables" properly?
Also, I'm not sure I understand how cache coherence would be relevant in the context of what the article is discussing, querying across all game entities to answer complex questions.
Wouldn't cache coherence matter more in the case of single object access?
The query engine works on top of an Entity Component System (ECS) storage that stores entities and their data in cache coherent (SoA - https://en.wikipedia.org/wiki/AoS_and_SoA) data structures.
With any kind of database - even in-memory SQLite - there is a query interface between you and the data. In ECS you access data directly, which is at least an order of magnitude faster.
Well, yes. Anything with "stats" or "inventory" looks very much like a database to me. This kind of thinking could also help avoid the risk of object-duplication bugs that occur in games which haven't been entirely rigorous about transactions.
Like the good old Dark Souls glitch that lets players drop a negative amount of items or consume a single item multiple times if triggered from some graphics options menu.
Any time you put data somewhere, organize it and give it an interface for efficient retrieval you have a database. Game state, generic data structures and practically an organization of data in software can be thought of the same way, so this is not exactly any sort of revelation.
The thing about a "database", relative to any other assembly of data, is that it typically has a universal query system whose particulars are not dependent on the schema of the data you're querying over. This is pretty much the focus of this article.
This brings back the memory of how we used Microsoft Excel to generate the .csv files to drive the game engine in Diablo 2 and stored the .xls files in version control, by the time we started work of production on the expansion we only needed about 3 of the 10 programmers (at our studio) to support the work because most of changes could be done by adding lines to the spreadsheets. This might sound rote today but the original, Diablo 1, had a bunch of hardcoded tables in the source code. I still used the Diablo 1 source code as a starting point for a lot of stuff, though. :)
Game designers really like Excel —for good reason.
I’ve set up the tech for many games over the decades where we had designers write tiny snippets of code in the cells of Excel spreadsheets. First in a custom assembly-like language (for an N64 game) then in SmallC, actual C++ and mostly in Lua.
The general theme was to use the grids to define state machines. Rows are states. Columns represent events. Cells define what to do at the intersection of an event in a certain state.
Later games also used Lua code cells as active spreadsheets in the game. The Lua statements would calculate results when queried. The charts didn’t just know about raw numbers and strings. They also could load and manipulate assets in the game. So, a designer could describe which UI asset to put next to the health bar based on the player’s current health stat without bothering a programmer or artist.
In the last case, we shipped two very different 3D mobile games using the same executable but different assets. Live hot-reloading of the Lua charts, the UI XML and all the other assets meant that we didn’t even need to bother making an editor for our custom engine. A good text editor window next to the game window was productive enough.
As part of the push to involve the designers more in production, I made a scripting engine and language to drive the quests in Diablo 2 roughly as you describe. By the time we go close to the end, none of the designers bothered to use it (we officially only had one at our studio but many people dabbled) so for my convenience I just took it out and replaced it with a function table that was there originally, I really regret that because I imagine the unofficial mod scene would be even more amazing if I left it in. The objects and monsters worked roughly as you describe, adding new ones (without new features) meant just adding and customizing a new line in the appropriate spreadsheets.
That's the dream! Live hot-loading and a text file. I was working on a Lua game a while back, and the feedback loop, although way faster than enterprise development, still got me bogged down when it came to balancing. I was using hardcoded tables for unit stats, and it's a surprising amount of mental overhead to go back and forth tweaking those.
Here's more useless trivia, my brain is cluttered with superfluous memories, I can't remember every algorithm but I remember the minutiae. I remember our producer Matt Householder telling me the bad experiences they had outsourcing the development of Diablo 1 expansion and ports meant they really wanted to make the Diablo 2 expansion in house. So at one point early in production on Diablo 2, our lead programmer, Rick Seis, had the .csv importing functionality as his primary task which seemed kind of mundane even at the time and now probably standard in most language libraries, and we transformed the Diablo 2 hard coded tables ( we were mostly following Dave Brevik's examples in Diablo 1!) into Excel spreadsheets. Then afterwards Peter Hu was on an optimization crusade towards the end, he made a .csv binary compacter/reader to save runtime.
Thanks, I did a quick search and found alot from whom should be considered the uber sources of information about Diablo development, David Brevik and Erich Shaeffer. I was just in the trenches for Diablo 2. One thing I remember about the crunch per David's piece on crunch, he neglected to mention that when he gave a speech announcing we would crunch until we finished, and if we finished in time for holiday release that year the entire studio would get a free trip to Las Vegas, or we would get fired. Kelly Johnson, a character artist was standing near me during the meeting and mumbled something under his breath but I don't think anyone thought we would generate the content required for that deadline, but I believe he had to give a speech about it I'm guessing for appearances' sake.
There are serious aesthetic ramifications for running with the idea of games (and especially game rules) as being primarily composed of databases.
Specifically, at least in my experience, foregrounding database thinking when making game rules tends to make it easier for designers to add more rules and rule variations, or a lot more "content" generally, that tends to be more shallow and less novel in terms of surprising interactivity.
Now, that can be a desired kind of game design! If you look at something like, say, Gran Tourismo, where the pleasure is in having hundreds of real cars modelled, that can be really enjoyable to a certain kind of player. And of course most static level data is just a giant collection of non-interactive data variations as well. There are certainly other examples.
But if you sit down with, say, the original NES Legend of Zelda, with a paper notebook in your lap, and every time you encounter a new enemy, you write down the enemy's name, what is interesting about them, and what you might have to do to implement them, what you're going to notice is that the lion's share of enemies have custom behavior and interactivity that will need to be special cased, and that that is all the meaningful work of implementing that enemy.
Now, obviously, you COULD still use something like a database as the central repository for distinguishing the identities of all of the monsters and their properties. But in practice, the property of "it gets on top of you and eats your magic shield" is only used by the Like-Likes, the property of "it grabs you and drags you back to the dungeon start" is only used by the wall hand guys, and on and on and on, and _aesthetically_ the game benefits from the fact that each interesting enemy property is only used by each unique enemy (or so I would very, very strongly argue).
I've sat down and performed that "notebook in the lap" exercise with a bunch of games in the past, with Half-Life, Castlevania:Symphony of the Night, and Mario 64 being particularly fruitful (the same exercise works for items and weapons in a game, and interactive objects in game levels as well). One of the big traits that makes SOTN the game that it is is that a fair number of the weapons and items in the game have all sorts of strange, surprising, extremely special case game code, like the Shield Rod.
My personal experience from working on game dev teams as a game programmer/designer is that, as game programmers are drawn more into database thinking as the lens for expressing game design, the kinds of sparkly, jagged, surprising rules I just gestured at start feeling more and more like violations of the architecture of the system, gumming it up and making it ugly, instead of the actual wonderful desired point of the entire enterprise of game making.
And so instead you get games where much of the variations between items or weapons or enemies are things like statistic percentage variations - this weapon has +20% critical hit chance compared to that weapon, this enemy absorbs that kind of damage, this weapon has that attack speed, this enemy has that running speed, this enemy can or can't throw grenades, and so on.
I feel like I encounter this kind of design a lot in more recent games made by large teams. And it makes sense. Often management needs tighter control over game rule possibilities because they have large teams and novel surprising custom game rules are legitimately unpredictable and hard for teams to control or reason about - especially in the context of long-lived games that are going to be maintained by lots of random people coming and going over a long time frame. Having weapons/items/enemies that vary only by properties make it easier to add or remove them to hit deadlines without breaking the critical path of the game, they're much easier to apply analytics to for balancing, and they're much easier to use as DLC without affecting games in potentially show stopping ways, too.
But there are absolutely aesthetic ramifications to this approach. At any given moment, when a player is playing a game and deciding, without consciously recognizing it, whether they're going to continue sticking with a game or whether they're bored, the question of what new kinds of new stuff they might still encounter if they keep playing is definitely a factor. And what kinds of rules a game designer can and does vary heavily affects that.
I suppose the antithesis of this is Nethack, where there is a huge variety of weird and wonderful interactions between items and the player, all of which are special-cased in the (C) source.
I have no expertise in this field but from what I gather many of these special cases (like the Like-Like eating your equipment) is a new component type on the enemy entity in an ECS architecture.
Of course, a game is a virtual world, and the player's behavior in this virtual world can express his inner thoughts to a certain extent, including good and malice, and some sociologists have conducted sociological research through the player's behavior in games.
Been playing Dyson Sphere Program lately. It's quite well-optimized (not quite MGS5-level, but pretty damn good!). Would love to know what performance savers they do under the hood. My guess is they are hard in the data-driven direction.
When I first tried to learn ECS (recently!), I couldn't wrap my mind around how arcane it seemed to be. What "components" should there be? What properties should each Component track? How do I organize it? What if I move things around? I have to write handlers that load and unload the damn things, and a System that somehow knows where to find certain properties on the various Components enough to operate on them? Examples were always pretty esoteric or contrived. Didn't help that they were all written in C++.
One day it clicked that this revolutionary idea sure looks a lot like a series of queries/views ("Systems") on a bunch of database tables ("Components") foreign-keyed to a single object ID ("Entities").
I once wrote an ECS for a game and it was literally a database without WHERE clauses (write your own if statements, indexes are too expensive). Each component was a table and your systems used left joins or inner joins to query the tables. The join algorithm was quite simple as well. Just a binary search to find the next element by its ID.
When you reinvent the wheel you can't call it the wheel or people will know it isn't new. 'Entity component' makes no sense for being a kind of spreadsheet, but at the very least it is a good approach to the problem. It should have been called 'state tables' or something like that.
When you open a container in the game, and it doesn't have a specified set of contents, the contents get loaded by picking randomly from a category linked to in the database. If, in one of the db tools, you follow the link from the container you can see what objects could load in it and by what likelihood. You can then follow those links through the db over to the object itself, which is also made up of db entries.
From the scripting system you can access the db for some amount of functionality. Their db doesn't contain the models or textures or strings. But it does contain the location of every tree in the game. If you wanted a mod with a dozen golden crates to go find, you would use the db system for the logic.