A bug fix is not always a breaking change but a breaking change is always a breaking change. SemVer doesn't care if it is a bug fix or not. The only single relevant detail is if it broke the behavior of the API for any reason. The reason for the breakage is entirely and completely irrelevant because the only relevant detail in SemVer is if it broke the public API. It's a binary "Yes/No" question.
Doesn't matter if it breaks it in a way that you don't expect users to have been using it - you broke it. Doesn't matter if you were just fixing a typo that changed behavior - you broke it. Doesn't matter if you were fixing a logical error - you broke it. Doesn't matter if you fixed the PRNG causing some results to no longer be possible - you broke it. Doesn't matter if you deleted a deprecated endpoint - you broke it. '
SemVer isn't asking why you broke it - it's asking did you break it.
Doesn't matter if it breaks it in a way that you don't expect users to have been using it - you broke it. Doesn't matter if you were just fixing a typo that changed behavior - you broke it. Doesn't matter if you were fixing a logical error - you broke it. Doesn't matter if you fixed the PRNG causing some results to no longer be possible - you broke it. Doesn't matter if you deleted a deprecated endpoint - you broke it. '
SemVer isn't asking why you broke it - it's asking did you break it.