Bayesianism is not wrong (it more or less can't be), the main problem is exactly what you mean by "knowing" and "entropy". Obviously, an embedded agent knowing from within physics is limited, but that's not a problem with theoretical Bayesianism, and it's not a problem if you're dealing with cards.
Wrong when applied in the general context of fundamental physics ?? (See also how entropy can often be approximated as an objective and property of a system, its density intensive, but in the general case that's wrong.)
Speaking of, I'm somewhat puzzled, you said that "under strict Bayesianism knowledge can't ever decrease", but how does it then deal with something as simple as card shuffling ??
Depending on how you want to do it, the hypotheses that get probabilities can be represented as computer programs that predict the input the agent receives. If done correctly (perfectly if you want it to be absolute) the probability-weighted sum of the calibration metrics won't ever go down.
I... am not sure that I understand what you mean ?
That after shuffling you know less about the stack of cards (that used to be at least partially revealed) is a fact that our model must follow or fail at being relevant.
It's been hard to find out more information about this, but I did find some :
Same thing : "And the Bayesian Second Law (BSL) tells us that this lack of knowledge — the amount we would learn on average by being told the exact state of the system, given that we were using the un-updated distribution — is always larger at the end of the experiment than at the beginning (up to corrections because the system may be emitting heat)."
Though I also did find another interesting thing :
"in a strict frequentist view, it is meaningless to talk about the probability of the true flux of the star: the true flux is (by definition) a single fixed value, and to talk about a frequency distribution for a fixed value is nonsense"
But that's pretty much the case in statistical physics ! A macrostate is actually NOT a state (=microstate), but a probability distribution !
"For Bayesians, probabilities are fundamentally related to our own knowledge about an event. This means, for example, that in a Bayesian view, we can meaningfully talk about the probability that the true flux of a star lies in a given range."
So looks like statistical physics are already at least part-way between Frequentist and Bayesian ??
Also, this one sounds potentially interesting, but sadly, paywalled...
http://www.av8n.com/physics/thermo/entropy.html#sec-s-not-kn...
http://www.av8n.com/physics/thermo/entropy-more.html#sec-s-c...
http://www.av8n.com/physics/thermo/s-relevance.html#ch-s-rel...