Strictly speaking, it should be a mistake to assign a probability equal to zero to any moves, even for illegal board moves, but especially for an AI that learns by example and self-play. It never gets taught the rules, it only gets shown the games -- there's no reason that it should conclude that the probability of a rook moving diagonally is exactly zero just because it's never seen it happen in the data, and gets penalized in training every time it tries it.
But even for a human, assigning probability of exactly zero is too strong. It would forbid any possibility that you misunderstand any rules, or forgot any special cases. It's a good idea to always maintain at least a small amount of epistemic humility that you might be mistaken about the rules, so that sufficiently overwhelmingly strong evidence could convince you that a move you thought was illegal turns out to be legal.
Every so often, I encounter someone saying that about some topic while also being wrong.
Also, it took me actually writing a chess game to learn about en passant capturing, the 50 moves without capturing or pawn move forced draw, and the 3 state repetition forced draw.
That's exactly right. A probability of zero is a truly absurd degree of self-confidence. It would be like if someone continued to insist that en passant capturing is illegal, even while being told otherwise by official chess judges, being read multiple rulebooks, being shown records of historic chess games in which it was used, and so on. P=0 means one's mind truly cannot be changed by anything, which leaves one wondering how it got to that state in the first place!
Probably most of us even know about en passant, so we think we know everything. But if I found myself in that same bewildering situation being talked down by a judge after an an opponent moved their rook diagonally, I'd have to either admit I was wrong about knowing all the rules, or else at least wonder how and why such an epic prank was being coordinated against me!
But the topic is chess, which does have a small number of fixed rules. You not knowing about en passant or 3 state repetition just means you never bothered to read all the rules. At some point, an LLM will learn the complete rule set.
> At some point, an LLM will learn the complete rule set.
Even if it does, it doesn't know that it has. And in principle, you can't know for sure if you have or not either. It's just a question of what odds you put on having learned a simplified version for all this time without having realised that yet. Or, if you're a professional chess player, the chance that right now you're dreaming and you're about to wake up and realise you dreamed about forgetting the 𐀀𐀁𐀂𐀃:𐀄𐀅𐀆𐀇𐀈𐀉 move that everyone knows (and you should've noticed because the text was all funny and you couldn't read it, which is a well-known sign of dreaming).
That many people act like things can be known 100% (including me) is evidence that humans quantise our certainty. My gut feeling is that anything over 95% likely is treated as certain, but this isn't something I've done any formal study in, and I'd assume that presentation matters to this number because nobody's[0] going to say that a D20 dice "never rolls a 1". But certainty isn't the same as knowledge, it's just belief[1].
[0] I only noticed at the last moment that this itself is an absolute, so I'm going to add this footnote saying "almost nobody".
[1] That said, I'm not sure what "knowledge" even is: we were taught the tripartite definition of "justified true belief", but as soon as it was introduced to us the teacher showed us the flaws, so I now regard "knowledge" as just the subjective experience of feeling like you have a justified true belief, where all that you really have backing up the feeling is a justified belief with no way to know if it's true, which obviously annoys a lot of people who want truth to be a thing we can actually access.
Say a white rook is on h7 and a white pawn is on g7.
Rook gets taken, then the pawn moves to g8 and promotes to a rook.
The rook kind of moved diagonally.
"Ah, when the two pieces are in this position, if you land on my rook, I have the option to remove my pawn from the board and then move my rook diagonally in front of where my pawn used to be."
There's got to be a probability cut-off, though. LLMs don't infinitely connect every token with every other token, some aren't connected at all, even if some association is taught, right?
The weights have finite precision which means they represent value-ranges / have error bars. So even if the weight is exactly 0 it does not represent complete confidence in it never occurring.
When relationships are represented implicitly by the magnitude of the dot product between two vectors, there's no particular advantage to not "creating" all relationships (i.e. enforcing orthogonality for "uncreated" relationships).
That’s not right; there are many vectors that go unbuilt between unrelated tokens. Creating a ton of empty relationships would obviously generate an immense amount of useless data.
Your links are not about actually orthogonal vectors, so they’re not relevant. Also that’s not what superposition is defined as in your own links:
> In this paper, we use toy models — small ReLU networks trained on synthetic data with sparse input features — to investigate how and when models represent more features than they have dimensions. We call this phenomenon superposition
Strictly speaking, it should be a mistake to assign a probability equal to zero to any moves, even for illegal board moves, but especially for an AI that learns by example and self-play. It never gets taught the rules, it only gets shown the games -- there's no reason that it should conclude that the probability of a rook moving diagonally is exactly zero just because it's never seen it happen in the data, and gets penalized in training every time it tries it.
But even for a human, assigning probability of exactly zero is too strong. It would forbid any possibility that you misunderstand any rules, or forgot any special cases. It's a good idea to always maintain at least a small amount of epistemic humility that you might be mistaken about the rules, so that sufficiently overwhelmingly strong evidence could convince you that a move you thought was illegal turns out to be legal.