I believe choosing a well known problem space in a well known language certainly influenced a lot of the behavior. AIs usefulness is correlated strongly with its training data and there’s no doubt been a significant amount of data about both the problem space and Python.
I’d love to see how this compares when either the problem space is different or the language/ecosystem is different.
I think you are correct. I work in game dev. Almost all code is in C/C++ (with some in Python and C#).
LLMs are nothing more than rubber ducking in game dev. The code they generate is often useful as a starting point or to lighten the mood because it's so bad you get a laugh. Beyond that it's broadly useless.
I put this down to the relatively small number of people who work in game dev resulting in relatively small number of blogs from which to "learn" game dev.
Game Dev is a conservative industry with a lot of magic sauce hidden inside companies for VERY good reasons.
One of my test queries for AI models is to ask it for an 8 bit asm function to do something that was invented recently enough that there is unlikely to be an implementation yet.
Multiplying two 24 bit posits in 8-bit Avr for instance. No models have succeeded yet, but usually because they try and put more than 8 bits into a register. Algorithmically it seems like they are on the right track but they don't seem to be able to hold the idea that registers are only 8-bits through the entirety of their response.
Can you generate 8-bit AVR assembly code to multiply two 24 bit posit numbers
You get some pretty funny results from the models that have no idea what a posit is. It's usually pretty clear to tell if they know what they are supposed to be doing. I haven't had a success yet (haven't tried for a while though). Some of them have come pretty close, but usually it's the trying to squeeze more than 8 bits of data into a register is what brings them down.
Yeah, so it’d be interesting to see if provided the correct context/your understanding of its error pattern, it can accomplish this.
One thing you learn quickly about working with LLMs if they have these kind of baked-in biases, some of which are very fixed and tied to their very limited ability to engage in novel reasoning (cc François Chollet), while others are far more loosely held/correctable. If it sticks with the errant patten, even when provided the proper context, it probably isn’t something an off-the-shelf model can handle.
> Although in fairness this was a year ago on GPT 3.5 IIRC
GPT3.5 was impressive at the time, but today's SOTA (like GPT 5 Pro) are almost night-and-difference both in terms of just producing better code for wider range of languages (I mostly do Rust and Clojure, handles those fine now, was awful with 3.5) and more importantly, in terms of following your instructions in user/system prompts, so it's easier to get higher quality code from it now, as long as you can put into words what "higher quality code" means for you.
I write Haskell with Claude Code and it's got remarkably good recently. We have some code at work that uses STM to have what is essentially a mutable state machine. I needed to split a state transition apart, and it did an admirable job. I had to intervene once or twice when it was going down a valid, but undesirable approach. This almost one shot performance was already a productivity boost, but didn't quite build. What I find most impressive now is the "fix" here is to literally have Claude run the build and see the errors. While GHC errors are verbose and not always the best it got everything building in a few more iterations. When it later got a test failure, I suggested we add a bit more logging - so it logged all state transitions, and spotted the unexpected transition and got the test passing. We really are a LONG way away from 3.5 performance.
Yeah, 3.5 was good when it came out but frankly anyone reviewing AI for coding not using sonnet 4.1, GPT-5 or equivalent is really not aware of what they've missed out on.
Yah, that’s a fair point.
I had assumed it’d remain relatively similar given that the training data would be smaller for languages like Haskell versus languages like Python & JavaScript.
Post-training in all frontier models has improved significantly wrt to programming language support. Take Elexir, which LLMs could barely handle a test ago, but now support has gotten really good
I am just not having this experience of AI being terribly useful. I don’t program as much in my role but I’ve found it’s a giant time sink. I recognize that many people are finding it incredibly helpful but when I get deeper into a particular issue or topic, it falls very flat.
This is my view on it too. Antirez is a Torvalds-level legend as far as I'm concerned, when he speaks I listen - but he is clearly seeing something here that I am not. I can't help but feel like there is an information asymmetry problem more generally here, which I guess is the point of this piece, but I also don't think that's substantially different to any other hype cycle - "What do they know that I don't?" Usually nothing.
A lot of AI optimist views are driven more by Moore's law like advances in the hardware rather than LLM algorithms being that special. Indeed the algorithms need to change really so future AIs can think and learn rather than just be pretrained. If you read Moravec's paper written in 1989 predicting human level AI progress around now (mid 2020s) there's nothing about LLMs or specific algorithms - it's all Moore's law type stuff. But it's proved pretty accurate.
- Today, AI is not incredibly useful and we are not 100% sure that it will improve forever, specially in a way that makes economic sense, but
- Investors are pouring lots of money into it. One should not assume that those investors are not making their due diligence. They are. The figures they have obtained from experts mean that AI is expected to continue improving in the short and medium term.
- Investors are not counting on people using AI to go to Mars. They are betting on AI replacing labor. The slice of the pie that is currently captured by labor, will be captured by capital instead. That's why they are pouring the money with such enthusiasm [^1].
The above is nothing new; it has been constantly happening since the Industrial Revolution. What is new is that AI has the potential to replace all of the remaining economic worth of humans, effectively leaving them out of the economy. Humans can still opt to "forcefully" participate in the economy or its rewards; though it's unclear if we will manage. In terms of pure economic incentives though, humans are destined to become redundant.
[^1]: That doesn't mean all the jobs will go away overnight, or that there won't be new jobs in the short and medium term.
I’m genuinely curious on the point about reducing headcount because AI will be more efficient. I’ve seen it articulated here but other places too that a company will be able to have less engineers because each would be more productive. What if companies kept the same number of people engineers but now massively out produce what they used to? And I disagree with the example that this is like typewriters replacing typists. I think typists have a fixed number of things that need to be typed. Software is different - a company that has a better or more feature rich project could gain on their competitors.
Curious if anyone else thinks this. Maybe it’s just optimism but I’ve yet to be convinced that a company would want to maintain its productivity through trading engineers for AI if it had the same opportunity to grow its productivity through AI and maintaining headcount.
And to add-on, isn’t there some market dynamics we are avoiding here with this example? If I’m an AI company and really produced a principal level engineer, why would I sell it for less than the labor market is willing to bear? Wouldn’t I price it perhaps less than the market but not so dramatically less as to lose money.
You make a good point. The shrinking headcount is not necessarily tied to mass-firing. It's more likely tied to +10M newly trained engineers entering the job market every year, but only 50 positions being opened.
Over time, with each recessions, headcount will shrink at some companies, and will not grow back to pre prior levels. Over time, the line trends downwards
Additional Context: The Getty Villa which is on the border of Malibu and Pacific Palisades was the structure that was threatened by fire directly. This article is about the Getty Center which lies in Brentwood and fires did not reach it.
Center: 1200 Getty Center Drive, Los Angeles, CA 90049
Villa: 17985 Pacific Coast Highway, Pacific Palisades, CA 90272
The reason this article is likely appearing now is because the Getty Center proper is currently in a zone which is under evacuation orders:
"The Getty Center, situated in Brentwood, draws 1.8 million visitors annually and houses hundreds of centuries-old art pieces from renowned artists such as Van Gogh, Rembrandt and Monet.
But even though as of Saturday, the center was included in a mandatory evacuation zone as a result of the Palisades Fire expansion into Brentwood, the center insisted its campus is the "safest place possible" for its massive art collection."
The Getty has an endowment in the billions and an entire team devoted to this that is permitted to be on site, coordinates with local fire and police services, etc. The entire center is also built into the hillside, with fireproof vaults in the underground levels, so there is no real risk to any of the collections.
An observation on how this has become extremely common nowadays.
Here, at work, in real life. People just argue with whatever dumb thing they can come up with, for the sake of arguing, it makes them feel smart. It's really hard to have a meaningful conversation with them.
I go to a couple philosophical discussion groups and the occasional town hall meeting. People just can't get their imaginary needs satisfied.
"But that area seems unsafe"
"We could hire a security guard to be around"
"But what if the security guard is a criminal, like in that one episode of muh favorite tv show"
"We could do an extensive background check, work with companies that have a good reputation, ..."
"But what if they make all that up, I saw that in a movie"
And nothing. ever. gets. done.
Btw, I've even seen people get a small round of applause by their peers after making one of such arguments irl. This comes to mind: https://www.youtube.com/watch?v=kn200lvmTZc.
Yes: you can notice this, for example, in the announcements they put out while the area around the Getty Villa evacuated and then burned, where they pointed out that non-emergency staff had evacuated. Similarly, I think JPL always had (emergency) staff present, and their own firefighting resources.
Having heard about wildfire policies for some high-profile institutions in fire-prone areas, they'll often have their own procedures, in coordination with local authorities, which may not involve evacuating when others do, and may involve people coming to the site when others are evacuating.
I imagine they have a number of life-support systems to ensure the staff can hang out in the building for a while in case of a severe natural disaster.
I don't think I've personally ever been as disillusioned with technology more than with the recent rapid adoption of AI.
I think AI is massively negative on the environment via increased energy consumption to train large LLMs, on creative endeavors via the auto-generated art, etc. on labor via automation of jobs through AI agents, or the degradation of the job market, on public communication via the proliferation of fake videos/news, on personal relationships via AI significant others, on child development through easier cheating, etc, etc.
I get that there's a certain inevitability about technology. I also feel - and this could be a terrible analogy - that might have been what it was like to develop nuclear weapons. Better to develop and harness the technology for yourself if your competition is doing so too. That's the world and the market that we've constructed and it's rational to take advantage where you can.
I certainly don't pretend to stand on any moral high ground about what I would do if given the opportunity to cash out from AI skills if I had them.
But it all makes me very sad. I think I'll go read a book.
As an SRE who’s moved between a few companies recently, I’ve noticed a trend towards simpler infrastructure options rather than k8s. Most places I interviewed at used AWS with just fleets of EC2s managed through ASG and instance refresh.
Deployments are no downtime. It’s pretty nice and if you want to run with docker on the host instances you can but you can also just install the application using systemd.
How do you build ec2's? Funny thing is, everyone will have their own answer. Few will look alike. It's maybe simpler, but also bespoke.
There's different definitions of simpler. I've rarely felt like folk's have a great grasp when they cobble together a bunch of options. Things that maybe one engineer started as a simple project 5 years ago grow and sprawl. They rarely stay simple.
What this team gets out of the box here is not to be trivialized. You havent started talking to the ci/cd that ArgoCD deploys. How are load balancers setup & managed, and at what price (Traeffik here is basically free)? Well managed Postgres with backups & replication, Lets Encrypt... handled. Is it really simpler to keep DIY'ing bits and bobs? Are you confident your simple start will work reliably & won't need improvement? What is better about your pile "simple" bespoke solutions, and why not have a platform that integrates varying concerns with consistent management patterns & autonomic operators?
EC2s are provisioned with Terraform or if you want to bypass that, click-ops. Nothing custom is needed. You can define a small shell script that is known as userdata that runs when the EC2 boots. That's standard and used everywhere that EC2s are used.
If you want more customization - again not necessary - you can build via Packer. The thing is, you have to do that anyways with Docker or whatever your container runtime happens to be. Docker images need patches as well if you install dependencies like openssl, libpql, etc.
ArgoCD - ArgoCD is great but it's something that has to be separate managed and comes with a database, UI layer, etc. and it's own set of permissions. The whole point of ArgoCD is to have deployments specifically around GitOps for container-based applications. Unnecessary if you don't have K8s.
Load balancers are simpler with ALBs and route directly to the auto-scaling groups. Traeffik is something that you need to manage. Also, K8s typically requires the use of external load balancers to work in the cloud as it is which means there's a layer of complexity that was introduced. Typically, K8s through the External Load Balancer Controller will provision the LB and attach targets. So it adds complexity. Traeffik isn't free, you are paying for the compute in your cluster which has the overhead of the control plane.
Let's Encrypt is another layer of complexity that has to be managed. AWS has ACM which can dynamically create and renew your certifications without you having to do anything.
K8s as a cluster needs to be managed and introduces significant breaking API changes across minor versions (1.23-1.24, etc.). Postgres managed as a stateful application also breaks some of the benefits of K8s in that it needs to be managed as a StatefulSet rather than as a ReplicaSet which means you likely need different deployment mechanisms to avoid destroying it accidentally.
Ironically, this architecture doesn't need any customizations at all. You can provision it with a few modules in Terraform or you can click ops the entire thing. K8s on the other hand, requires pretty constant attention. You've now introduced the management of the cluster, the building of the image, management of ArgoCD, Traeffik as well as external load balancers.
Basically:
- Route53 for domain management
- ACM for cert management
- EC2s for LBs
- ASGs to manage EC2s
- Small shell script within userdata to configure the host vm
- RDS if you need a database
Avoiding the operational and actual costs of K8s.
I think that there are compelling reasons to use K8s:
- Lots of smaller services that need to interface with each other
- Product is deployed to multiple clouds or in a hybrid setting
This is what we do for GCE. Our CI builds a system image and then that's deployed across our datacentres. Worked perfectly for years, but still people want to switch to kubernetes. What can you do?
Oh definitely that happens and from my experience I don’t think there’s a lot that can be done unless there’s strong faith in engineering leadership to push back on the complexity.
> Serve traffic behind a load balancer that has a WAF
whats the cheapest non aws way to do this? cloudflare on everything? is there another option? just trying to learn whats out there. WAF mainly protects against ddos right?
There are also some projects like Coraza in the works: https://coraza.io/
Probably not what you're looking for if you want a cloud service to take care of everything for you, though, because of the question below (just thought that it might be useful to point out that anyone can run their own WAF if need be).
> WAF mainly protects against ddos right?
Typically WAF might be offered as a part of a larger cloud service that would include DDoS protection.
However, on its own, it is meant to filter traffic that might be harmful and attempt to exploit various vulnerabilities. A bit like an anti-virus in a sense, but for web requests. Some people argue that WAF solutions can be problematic because they encourage an attitude of "so what if there's a log4j vulnerability in the codebase, the WAF will take care of it" instead of making sure that the actual code is secure, but opinions are split there (defense in depth and the Swiss cheese model).
GP has some good suggestions. For implementation of these, Cloudflare is a decent first stop - though they are a little hostile to non-vanilla internet users. Their free plan offers sensible security (SSL termination, WAF, DDOS protection) out of the box, with a straight forward UI.
Network segregation for database (separate subnets) would be a config option wherever you're hosting (AWS/Google Cloud/etc.) said database/application.
It’s a feature of an LB that consolidates the actions of blocking ports except for the ones you are using, fail-fast on paths that scrapers tend to check (e.g. /wp-admin, /phpMyAdmin) so it doesn’t end up in normal request logging, set rate limits, fail-to-ban conditions, etc.
Seems like we’re recreating the Middle Ages financially a bit here doesn’t it?
Mobility is linked to previous generations more than your own. Wealth is more important than income because there are so few jobs that can actually build enough wealth to own a home.
I’d love to see how this compares when either the problem space is different or the language/ecosystem is different.
It was a great read regardless!