It matters because a good versioning system indicates to users what to expect from the release. If version numbers are major.minor.patch, then incrementing the patch number should indicate that you've only fixed bugs and it should be safe to install without risking new bugs. Incrementing the minor number should mean you've made feature changes but everything is backwards compatible. Incrementing the major number should mean you've made major changes that break compatibility.

See http://semver.org/ for more details.

Edit: others have pointed out that this is more important for libraries and less so for user-facing applications. For user-facing applications I'd recommend two version numbers so you can distinguish between feature releases and bugfix-only releases, which can be a useful indicator to the end user.