My niche is in video game programming, and I am very specialized in a specific area. So I might ask things like how would one architect a certain game system with a number of requirements, to meet certain player expectations, and be compatible with a number of things.

Unfortunately, it hasn’t been helpful once, and often due to the same reason - when the question gets specific enough, it hallucinates because it doesn’t know, just like in the early days.

Moreover, I am a domain expert in my area, so I only ask for help when the problem is really difficult. For example, when it would take me several days to come up with an answer and a few more weeks to refine it.

Game development has a lot of enthusiasts online sharing material, but most of this material is at junior to intermediate level. You very quickly run out of resources for questions at a principal level, even if you know the problems you have have been solved in other AAA companies.

You have to rely on your industry friends, paid support from middleware providers, rare textbooks, conferences, and, on the off-chance that anything useful got scooped up into the training data set - GPT. But GPT has been more like wishful thinking for me.

Interesting. I also work in game development, and I tend to work on project-specific optimization problems, and I've had the opposite experience.

If I have to solve a hairy problem specific to our game's architecture, obviously I'm not going to ask ChatGPT to solve that for me. It's everything else that it works so well for. The stuff that I could do, but it's not really worth my time to actually do it when I can be focusing on the hard stuff.

One example: there was a custom protocol our game servers used to communicate with some other service. For reasons, we relied on an open-source tool to handle communication over this protocol, but then we decided we wanted to switch to an in-code solution. Rather than study the open source tool's code, rewrite it in the language we used, write tests for it, generate some test data... I just gave ChatGPT the original source and the protocol spec and spent 10 minutes walking it through the problem. I had a solution (with tests) in under half an hour when doing it all myself would've taken the afternoon. Then I went back to working on the actual hard stuff that my human brain was needed to solve.

I can't imagine being so specialized that I only ever work on difficult problems within my niche and nothing else. There's always some extra query to write, some API to interface with, some tests to write... it's not a matter of being able to do it myself, it's a matter of being able to focus primarily on the stuff I need to do myself.

Being able to offload the menial work to an AI also just changes the sorts of stuff I'm willing to do with my time. As a standalone software engineer, I will often choose not to write some simple'ish tool or script that might be useful because it might not be worth my time to write it, especially factoring in the cost of context switching. Nothing ground breaking, just something that might not be worth half an hour of my time. But I can just tell AI to write the script for me and I get it in a couple minutes. So instead of doing all my work without access to some convenient small custom tools, now I can do my work with them, with very little change to my workflow.

>I can't imagine being so specialized that I only ever work on difficult problems within my niche and nothing else. There's always some extra query to write, some API to interface with, some tests to write... it's not a matter of being able to do it myself, it's a matter of being able to focus primarily on the stuff I need to do myself.

there might simply not be enough literature for LLM's to properly write this stuff in certain domains. I'm sure a graphics programmer would consider a lot of shader and DirectX API calls to be busy work, but I'm not sure if GPT can get more than a basic tutorial renderer working. Simply because there really isn't that much public literature to begin with, especially for DX12 and Vulkan. That part of games has tons of tribal knowledge kept in-house at large studios and Nvidia/intel/AMD so there's not much to go on.

But I can see it replacing various kinds of tools programming or even UI work soon, if not right now. It sounds like GPT works best for scripting tasks and there's tons of web literature to go off of (and many programmers hate UI work to begin with).

Well, I think most software engineers in games don’t work all that much with scripts or database queries, nor write that many tests for systems of scale that GPT could produce. You might be in devops, tools, or similar if you deal with a lot of that in game dev.

GPT code in a lot of critical path systems wouldn’t pass code review, not probably integrate well enough into any bespoke realtime system. It seems to be more useful in providing second opinions on high level decisions to me, but still not useful enough to use.

Maybe it could help with some light Lua or C# gameplay scripting, although I think co-pilot works much better. But all that doesn’t matter as due to licensing, the AAA industry still generally can’t use any of these generative AIs for code. Owning and being able to copyright all code and assets in a game is normally a requirement set by large publishers.

To conclude, my experience is indeed very different from yours.

I think the difference in our perspectives is the type of studios we work for. In a AAA studio what you're saying makes perfect sense. But I've worked entirely for small- and mid-size studios where developers are often asked to help out with things outside their specialization. In my world, even having a specialization probably means you're more experienced and thus you're involved in a variety of projects.

Whether that's "most" software engineers in games or not I can't say. AAA studios employ way more engineers per project but there are comparatively way more small- and mid-sized developers out there. It's interesting how strong the bubbles are, even within a niche industry like games.

I think GPT is comparatively poor at game dev due to a relatively small training corpus, with much more code being locked away in binaries (.uproject, etc), and game code rarely being open sourced

Godot might benefit more than other engines, since much of the code is stored as plaintext GDscript and makes it to GitHub more frequently

I'm interested to know if you've tried creating a custom GPT with their builder or the API. If you have enough old example code, notes, or those rare textbooks you mention you could add those as files and see if the built in RAG improves the answers it gives.

I tried building a custom GPT but the training data it has is not sufficient, no matter how well it’s steered.

Documents and code are confidential in the AAA games industry as they are the money makers. Developers are not free to hand them over to third parties, that would be known as a leak. With textbooks, that would be a pretty grey area use case. So I’ve not experimented with that.

I think it could help, but because it’s so infeasible practically, there’s no incentive to try this with synthetic data, too.

It struggles with (industrial, not hobbyist) embedded firmware a fair bit. I can almost coax decent results for simple tasks out of it, sometimes.

LLMs almost never write good senior quality code at first in niche disciplines. You need to finesse it a lot to have it produce the correct answer. And that makes it unusable for when you genuinely do not know the answer to the question you’re asking, which is kind of the entire point.