This completely ignores the fact that the chromatic note relationships in different keys are not the same. F# is not the same pitch as Gb. In fact, I find thinking about notes in terms of a moveable do solfege (not as static names at all) to be by far the most effective system, since it really manifests the relationships in the key that you're playing in. When you realize those functional relationships, along with the requisite tensions in pitch, which you can emphasize on a non-justified instrument, music making becomes a lot more fun and easy.
There's nothing special about sharps and flats in that regard. Depending on your tuning system C could be different in every scale. That doesn't make it something you want to encode in your notation.
Your point about the "white keys" is well taken, which is why I mentioned moveable do solfege. The absolute pitch of "C" takes on a different name in every key, which respects its relationship within that key.
However, there's a reason that scales like the melodic minor are written with sharps (or "sharped flats"/naturals) ascending and flats/naturals descending. Sharps emphasize rising pitch and flats emphasize falling pitch, and can thus be bent slightly higher and lower, respectively. You would lose that with fixed chromatic note names.
(And as an aside, recognizing when a note in a scale has been modified from its normal position, and in which direction it was modified, is yet another reason to practice your scales!)
Which is exactly my point. One size fits all doesn't work. The scale or notation system should be appropriate to the music. AFAIK, Indian classical has its own way of doing things, and most blues and jazz isn't notated the same way as classical music, either. Going to a fully justified/chromatically-based system is just sweeping the intricacies and nuances under the rug. Plus, naming "all twelve" tones doesn't even begin to address what you would do with microtonal music.
