I think this is a great question we should all think about for ourselves in advance - what does it have to do to convince you it’s actually intelligent.

Because once it does that thing without you having expressly decided that is the goal, it’s very tempting to just move the goal a liiiitle bit further away

Since when training and fine-tuning isn't learning? Individual sessions of LLMs are not learning, but models as products surely are - the feedback loop is just iterated manually.

Is the LLM learning, or are the humans involved in every stop of every iteration doing the learning?

To me the LLM loophole/"hack" closings just feel like a human vs human cat&mouse game with some Chat UI in the middle.

Yes, it is a collaborative endeavor, and the whole could be seen as a man-machine superorganism, or, more profoundly our own sense of separateness is illusory as we and the entire universe are one.

That the LLMs are actually evolving before my eyes within & across sessions, without human-in-the-loop "hand tuning" iterations (sounds like injections of glorified if statements to this guy) .

You want to witness the learning firsthand, I suppose. That's reasonable. I'd also suggest that it's possible to imagine questions for the LLM that it cannot solve today and that you reasonably believe will not be available to OpenAI to "hand tune" it against. If you can come up with such a problem, it can't solve it today, but does in the future then you have some evidence, I'd think.

What's more, is we can do that today. Just think of any problem which you suspect won't be included in OpenAI's hand-tunings and check both 3.5 and 4.