> Consider how ridiculous that would sound if you asked "What if one of your friends had emailed you their phone number and wanted to remove your access to it (or had emailed you and wanted to remove your knowledge of their email address)?

That would be a fine system -- where phone numbers and other means of contact can become invalidated if the owner wants them to.