Look at other popular libraries, and consider how they solve this problem - and whether it even is a problem for them (and if not, why).
Usually, this sort of versioning is done if and only if a breaking API change occurs, and such changes are avoided if possible for that very reason. If you only rev the major version number on breaking changes, then you only need to reflect that number in your API (and even then only if/when you actually rev it).
I an of the view that it is a problem, but one which is almost wholly ignored and as such, not addressed.
Generally, software projects version as you have described, based on breaking the external API. If the API is unchanged, the library is deemed to be effectively unchanged.
I am not happy with this. ANY change in a library requires revalidation - and so the only way to deal with this is to have no change at all, not even in the library binary. To do otherwise is to assume correctness, which as we know, given the nature of software, this is something which is never always true.
This level of care about change is I think almost never used and normally software libraries take for themselves the advantages of fixing versioning at the API-breaking level - it makes development easier, at some of cost to the users of reliability (if the users accept the changes without revalidation) or at some cost to the users of work (if the users revalidate).
I think a library should act as a factorization - it should take into itself all possible work which would otherwise have to be performed by ALL its users, so that that multiplicity of work is converted into a single instance of that work, performed by the library authors.
Usually, this sort of versioning is done if and only if a breaking API change occurs, and such changes are avoided if possible for that very reason. If you only rev the major version number on breaking changes, then you only need to reflect that number in your API (and even then only if/when you actually rev it).