These llms are Bayesian models. I’m not quite sure what the calculation would be but it should be possible to get a confidence score from some combination of each probability.
The problem is that it would be a confidence score that the words would follow each other in that order in a real text, which would be both way lower than the chances of them being correct (remember, the models are trained to reproduce their training samples), and also more correlated with what people were writing online than with the truth.
Sure but they’re correlated. I think a team like openAI could definitely come up with some kind of useful “confidence score” based on Bayesian probability scores plus some other metrics, even if they’ll never have 100% confidence
