The proposed attack on Apple's protocol doesn't work. The user's device adds randomness when generating an outer encryption key for the voucher. Even if an adversary obtains both the hash set and the blinding key, they're just in the same position as Apple—only able to decrypt if there's a hash match. The paper could do a better job explaining how the ECC blinding scheme works.
This is one of the concerns in the OP, have an AI generate millions of variations of a certain kind of images and check the hashes. In this case it boils down to how common false positives neural hashes are.
Given some CP image, an attacker could perhaps morph it into an innocent looking image while maintaining the hash. Then spread this image on the web, and incriminate everybody.
Yes perceptual hashes are not cryptographically secure so you can probably generate collisions easily, (i.e. a natural looking image which has a attacker-specified hash).
Sounds like a fantastic way for law enforcement to get into your phone with probably cause. Random message you a benign picture from some rando account with a matching hash. Immediate capture for CP, data mine the phone, insert rootkit, 'so sorry about the time and money you lost - toodles'.
Warrants do have to name why, and where. However, anything they find along the way is fair game. If they open your trunk to find drugs and see a dead body, then the dead body is still admissible. (Assuming that the opening the trunk for drugs is okay.)
It'd be interesting to see how the way common images are reused (for example in memes by only adding text) would be enough to change that hash. If it wasn't enough it could spread very quickly.
Of course I'd dare not research or tinker with it lest I'll be added to a list somewhere such is the chilling effect.
I guess in that case they'd delete that single hash from the database because they'd still have an endless (sadly) supply of other bad image hashes to use instead.
> Then spread this image on the web, and incriminate everybody.
You'd still have to generate several images and persuade people to download multiple of them into their photo roll. And as I understand it there's yet another layer of Apple employees to review the photo metadata before it ever makes its way to law enforcement.
That does seem like an interesting protest vector, though. Generate a bunch of images that match CSAM images but are mundane. Then have everyone download them and send them to their cloud. Someone then needs to spend resources determining that the images are _not_ actual matches. Basically, a DDOS attack on the functionality.
It's a risky bet, though: if somehow that intermediate layer fails and you find yourself locked up and accused of storing/disseminating CSAM material, it's not like the civil rights era when your friends and neighbors (and hopefully employers) will understand you've been arrested for a peaceful protest.
The smarter, if potentially less ethical solution is to encode such images and make memes with them. One of them going viral is likely to flag an enormous number of people along the way.
It won't be long until these type of systems are mandated. Combined with a hardware root of trust it's not inconceivable that modifying your hardware not to report home will also be made a crime. It never stops with CSAM either, pretty soon it's terrorism and whatever vague new definition they use.
The focus on CSAM seems extremely hypocritical when authorities make such little effort to stop ongoing CSA. I would encourage everyone to research the Sophie Long case. Unless there is image or video evidence the police make little effort to investigate CSA because it's resource intensive.
Total surveillance is definitely the end goal of policing forces. It's in their very nature of getting their job done (what better way to catch criminals than a computer constantly scanning every move of everyone) and why people need to always push back against these "think of the children" scapegoats they use to get their foot in the door and get more control.
> It never stops with CSAM either, pretty soon it's terrorism and whatever vague new definition they use.
But PhotoDNA has been scanning cloud photos (Google, Dropbox, Microsoft, etc.,) to detect CSAM content for a decade now and this "pretty soon it's terrorism" slippery slope hasn't yet manifested, has it?
If the slope was going to be slippery, wouldn't we have seen some evidence of that by now?
Don’t be so naive. It took less than 3 years for dns blocking to go from csam to copyright infringement. It always was about building censorship infra. I’ve been fighting internet censorship for over a decade and it only gets worse and worse every generation of technology. I want to throw all this government spyware away. Dystopia landed a long time ago.
Regardless of whether this attack works or not, you'd assume this scheme produces a wider attack surface against pictures in iCloud and against iCloud users. One attack I could imagine is a hacker uploading child porn to a hacked device to trigger immediate enforcement against a user (and sure, maybe there are more controls involved but would you carry around a very well-protected, well-designed hand grenade in your wallet just so you're bad, it'll explode).
In case you didn't the topic, what is specific (for now, for now...)to iCloud/apple is the "we're scanning your photos on your device and maybe reporting them if they're bad" approach. So you get the local hashes on the supposedly encrypted files and you get the situation of local files trigger global effects like the police swooping down and arresting you. So that's why despicable and hair-brained scheme in specific produces a greater "attack surface" in multiple ways.
And again, sure, Apple doing this quite possibly will set a precedent for Google et al to answer the other ambiguous meanings your ambiguous comment has.
Literally for almost every other big cloud provider. (Facebook, Instagram, Discord, Reddit, Twitter and so on.) Granting that you have access by phone.
Or even a hash collision with a banned image. Actually, if that could be generated this thing could fall apart pretty quickly if such collisions could be widely distributed.
For some reason, after reading the initial reporting on this system, I thought it was running against any photos on your iPhone, but now I read the actual paper, it seems like it only applies to photos destined to be uploaded to iCloud? So users can opt out by not using iCloud?
Much of the discussion is about how trivial it would be for Apple to start scanning any photos on the phone at a later date.
Right now they are able to bill this as doing what they currently do server side, but client side. Later, they can say they are simply applying the same "protections" to all photos instead of merely the ones being uploaded to iCloud.
They can do it already. System is full black box, and all we have is their word. So, saying that adding something might enable something else, is not strong argument.
By that logic we may as well never question anything any of these companies - or even governments really - do because they might just find a way to do it secretly and maybe nobody would ever figure it out.
Friendly reminder: until ios source code is closed all privacy claims is only backed by trust. They easily can do whatever they want if you're not compiling from source.
If Apple is to keep their word about guaranteeing the privacy of non-CSAM photos (which this whole discussion is about them not doing a very good job of), then they would only be able to do that with photos stored in iCloud because of this technical specification as to how the identification process works. That being said, other photos across your device are still monitored in a different way. For example Apple will scan photos that you send or receive via iMessage to automatically detect if they're nudes, and if you're underage, they will block them/send a notification to your parents.
As far as I know apple plans to put up 2 systems, one focused on phones of people age < 13 which filters "more or less" any photos and uses AI to detect explicit photos and one which looks for known child pornographic photos and for now seems to not necessary apply to all photos.
They have a system that checks for hashes of images to try and find specific CSAM from a database when images are uploaded to iCloud, this already happens but is now moving on device. When explaining this I've used the analogy that here they are looking for specific images of a cat, not all images that may contain a cat. When multiple images are detected (some threshold not defined) it triggers an internal check at apple of details about this hash and may then involve law enforcement.
The other one is for children 12 and under, that are inside a family group. The parents are able to set it up to show a pop up when it detects adult content. In this case they are looking for cats in any image, rather than specific cat image. The popup lets them know it may be an image not suitable for kids, that its not their fault and they can choose to ignore it. It also lets them know if they chose to open it anyway their parents will get notified and be able to see what they've seen.
Yeah pretty much. Another way of thinking about it, is that to upload an image to iCloud, your phone must provide a cryptographic safety voucher to prove the image isn’t CSAM.
The question presumes the database leak also comes with the server side secret for blinding the CSAM database, which is unlikely (that’s not how HSMs work) and would be a general catastrophe (it would leak the Neural Hashes of photos in the NCMEC database, which are supposed to remain secret).
Yeah, I've worked with HSMs in the past and to say that it's a challenge to get key material out of them is an understatement. That said, a lot of this depends on the architecture surrounding the HSM - if the key material leaves the HSM at any point, you've basically increased your attack surface from an incredibly secure box to whatever your surrounding interfaces are. At Apple's scale, I have to imagine it's more economical to have some kind of envelope encryption - maybe this is the right attack vector for a malicious actor to hit?
The question doesn't presume that, as the the secret for blinding the CSAM database would only be helpful if a third party were also looking to see which accounts contained CSAM.
In this case, the question assumes that an attacker would more or less be creating their own database of hashes and derived keys (to search for and decrypt known photos and associate them with user accounts, or to bruteforce unknown photos), and would therefore have no need to worry about acquiring the key used for blinding the CSAM hash database.
> What's to stop an attacker from generating a NeuralHash of popular memes, deriving a key, then bruteforcing the leaked data until it successfully decrypts an entry, thus verifying the contents within a specific user's cloud photo library, and degrading their level of privacy?
Decrypting vouchers requires the server blinding key and the NeuralHash derived metadata of the input image (technical summary page 10, Bellare Fig. 1 line 18). This attacker only has the latter.
> For CSAM matches, the cryptographic header in the voucher combines with the server-side blinding secret (that was used to blind the known CSAM database at setup time) to successfully decrypt the outer layer of encryption.
In the text you referenced, it specifically says that the blinding key would be needed to decrypt vouchers which are CSAM matches. This is because Apple set up their CSAM database in a blinded manner. Therefore to access a hash from the database from which to derive a decryption key, Apple would need the blinding key to first decrypt that hash value.
However, and attacker would be generating their own (presumably unblinded) database, and therefore wouldn't need to access Apple's blinding key.
I’m a little confused. The vouchers you are trying to decrypt have already been generated. How does it matter if the attacker can decrypt vouchers from a database they created but was not used by the vouchers in the breached data?
It is my understanding that the vouchers are only encrypted with a key derived from the NeuralHash of the photo. Therefore an attacker would only need to find a matching NeuralHash, to decrypt the voucher.
Apple needs the blinding key, because they encrypt their list of NeuralHashes hashes first, so that others cannot see exactly which CSAM hashes they're testing against. Therefore they first need to decrypt their own database in order to get the corresponding hash value from which to derive the decryption key.
That’s wrong. Dec(H′(\Hat{S}_j), ct_j) requires \alpha the server secret to determine the decryption key using Boneh’s notation of the PSI system. Or looking from the other direction, the encryption uses both w (NeuralHash) and L (\alpha G, for server secret \alpha).
I believe the math you outline above refers to this step, located on page 7 of the Technical Summary:
> Next, the client creates a cryptographic safety voucher that has the following properties: If the user image hash matches the entry in the known CSAM hash list, then the NeuralHash of the user image exactly transforms to the blinded hash if it went through the series of transformations done at database setup time. Based on this property, the server will be able to use the cryptographic header (derived from the NeuralHash) and using the server-side secret, can compute the derived encryption key and successfully decrypt the associated payload data.
Is that correct?
If so, then I agree that it is true that in the PSI system the server secret is completely necessary as part of the decryption process in order to decrypt the matching hash in the pointed-to location in the table. That being said, looking only at the information encrypted by the client, I don't think the server secret comes into play, right?
If I'm misunderstanding, and you're confident that an attacker would have to have the server secret to decrypt a photo (even if they already knew that photo's NeuralHash and were able to defeat the internal layer of encryption), then I definitely recommend posting a well-outlined answer to the Cryptography Stack Exchange, as that would be super helpful!
Yes the server secret comes into play looking at the encryption key on the client. This is the value L, which you can think of as a “public key” in usual ECC schemes.
It’s not useful to talk about defeating the “internal” layer of encryption separately from the “outer” layer because vouchers should be stored as generated by clients at rest. The database leak should not include any unwrapped vouchers (that would be like using a password hash but storing plaintext passwords anyways).
> However, the phrase “child pornography” is almost too sterile and generic to properly exemplify the horrors of what is being created. That is why many advocates, including the National Center for Missing and Exploited Children (NCMEC), believe this phrase to be outdated.
> NCMEC refers to these kinds of material as Child Sexual Abuse Material (CSAM), in order to “most accurately reflect what is depicted- the sexual abuse and exploitation of children”.
> As a result, many organizations and advocates now refer to this material by the new term rather than "child pornography" because it explicitly ties the material to the source of the problem; the abuse that is being perpetuated to create it. Furthermore, children are re-victimized every time a file is shared, sustaining the abuse in a continuous loop.
I realized child beauty pagents and other sexual exploitation of children would be considered Child Sexual Abuse Material in some cases. People, even those with good intentions, are not going to like where this is going.
'Child pornography' is already too long to say repeatedly in a sentence so it is often shortened to "child porn." CSAM has far too many syllables so it too is shortened to two syllables in the form of the spoken acronym "CSAM" and so we're back to square one.
With that said this is a fascinating display someone pressing the reverse button on the euphemism treadmill. I don't think I've ever seen that before.
I don't think the number of syllables is the important thing here. Rather that the term "porn" doesn't—by itself!—impute any moral judgement. It's just a thing. Some people think it's all bad, others think some is okay while some isn't, and some think anything with consenting performers is fair game.
But children are incapable of being consenting performers. That's what separates abuse material from porn. So to make damn sure there isn't any overlap in the Venn diagram of media that depicts sexual acts, they'd rather not associate "child porn" with anything else that exists in the universe of "porn".
It's a completely separate category, not some "bad" end of a spectrum.
That's the reasoning I've heard before, anyway.
And I agree with you on the reverse treadmill thing. It's interesting. On a related tangent: I've always hated how journalists use the term "sexual assault" to refer to a wide range of offenses, from forcible rape to a passing grope. Although those are both bad things, it's clear that one is tremendously more harmful than the other. We should use language to clarify that.
The number of syllables matter if the goal is to prevent a term from feeling too sterile. If a term is too long and unwieldy then people will naturally shorten it to an acronym. And once they're using an acronym then we're back to square one because acronyms are devoid of the meaning that words have.
I've wondered if there is an intent to distinguish between child pornography that teenagers are producing on their own to share with each other and the kind child pornography that pedophiles and child rapists create forcibly against the will of the children in the content.
Maybe that's the actual difference between CP and CSAM.
Maybe both are a subset of CP.
I think saying “child porn” is clearer. That term is horrific as it is. If anyone sees that in a sentence they’ll be rightly revulsed and know what’s being talked about right away.
I had to have someone explain the CSAM acronym. How many people are going to skip over that because they assume its something benign and unrelated?
You ever notice when cops bust a prostitution ring they never call it that? It's been renamed human trafficking. Law enforcement is all in on the marketing game.
Using scarier words will get the public to trade liberty for security every time.
Last year Sam Harris interviewed Gabriel J.X. Dance, the deputy investigations editor at The New York Times [0]. They speak to this. It's a tough episode to listen to - they cover many uncomfortable topics along this theme.
From what I recall of the episode, porn (rightly or wrongly) is seen as opt-in for the participants. But if children are involved it cannot be opt-in and is more appropriately described as rape or sexual abuse. Thus CSAM is the preferred term.
That's the CSAM part. The iMessages part uses a neural network to detect explicit photos through machine learning.
What happens when Apple, or the government, mandate an expansion of CSAM into detecting new material? Apple already has built a neural network to detect new explicit material...
Also, what happens when the USG mandates that Top Secret classified material must also be added to the database? Or when Russia mandates that homosexual pornography must be added to the database?
> Also, what happens when the USG mandates that Top Secret classified material must also be added to the database?
That doesn't appear to have happened in the 10+ years this kind of scanning has already been happening at various cloud providers - what specifically about Apple moving it down to the phone from iCloud Photos makes this more likely now?
Depends on the age of consent in both person’s jurisdiction and where they go with those pictures. My 16 year-old self got quite a lecture when my girlfriend and I took some pics of each other and my parents found them. (Age of consent was 13 where we lived, but it was 18 just up the road in another state).
I'm not well versed enough in the topic (or the law) to have answers. I do think the podcast episode I pointed to does give a lot of food for thought. For me it was an eye opener, as a software engineer trying to build a beautiful future with strong beliefs around e2e encryption, etc.
CSAM is not necessarily pornographic material, but merely material that becomes objectionable within context.
For example, parents' photos of kids in the bath isn't CP. However _someone else_ having a _quantity_ of bath photos is CSAM, if they have no reasonable reason for possessing them.
It’s very muddy, though. The number of pictures on one’s hard drive is irrelevant to the fact that a child has been abused or not. In your example, none of the children would have been abused. Also, who gets to define how much “a lot” is? We can’t say that it’s ok if it’s your children (or grand-children’s, or nephews, etc), because most of child abuse cases involve close family members or close friends.
We definitely should punish exploitation of children, including sexual, and we definitely should punish distributing images of this. But conflating exploiting, distributing, and viewing, and then putting a big taboo on this, is really not ideal. These things are different. Otherwise what we end up with is righteous frenzy when someone gets punished for sexting.
That is correct. The children within the images that are classified as CSAM don't necessarily have (though frequently are) to be abused.
For example, the NCMEC database contains hashes for Nirvana's Nevermind cover. Completely innocent to possess within it's original and intended context.
I have not said whether I agree with this, because I do see problems when automating the process.
However, the precedent for it being used as a flag for law enforcement has already happened. Having a collection of similar imagery is considered CSAM - and that is likely correct. A collection is probably not innocent. But having one or two images may happen incidentally without your awareness of it.
As to who decides what constitutes significance? That is where you'll hit the most problems, and reasonable discussion of it will be quickly shut down with the same arguments used for automating a flagging system. The conversation requires nuance, but those currently calling for such systems aren't interested in a good faith discussion.
> I have not said whether I agree with this, because I do see problems when automating the process.
The process will never be fully automated, regardless of what they say. Cases will need to be reviewed. Things will need to be checked at some point.
They are trying to play the cog in the machine, that mechanically transmits information to law enforcement. But if we’ve learnt anything the last decade is that cogs are not impartial and can be very dangerous, if only because of the scale at which they operate. I can see several ways a user can face a kafkaesque uphill battle to prove their innocence. In several countries, just a child pornography case can be a social death sentence. And even a fraction of a percent of mistakes will mean millions of people might be dragged in this (let’s not kid ourselves: this is never going to stay in the US).
Personally I think (what Apple is doing) is misguided and ripe for abuse. I am very disappointed that they, of all companies, are pushing this nightmare.
> As to who decides what constitutes significance? That is where you'll hit the most problems, and reasonable discussion of it will be quickly shut down with the same arguments used for automating a flagging system. The conversation requires nuance, but those currently calling for such systems aren't interested in a good faith discussion.
Ultimately, a tech company has no business making this sort of decisions. This is something that needs to be sorted out by law.
Unfortunately, a nuanced discussion is very unlikely these days. Anyone looking not agressive enough will be pilloried.
CSAM indicates you think the material, and the practice of making it, is abhorrent. "Child porn" is something some (severely deranged) people might actually want to see. The upshot of this is, if you put "ios child porn detection features" into Google, Google will see "child porn" and may think that's what you're searching for, put up warning banners reminding you that child porn is super illegal and that you should not be searching for it, and probably notify the FBI who will add you to a watchlist. If you put "ios csam detection feature" into Google, it helps Google know that you are not searching for child porn itself.
Social dynamics. The kind of person who calls CSAM CSAM is unlikely to favor it, let alone distribute it, so people interested in it are unlikely to search for it by that name.
Why does Apple even bother with encryption? They should just skip all of the warrant requirements etc and use their iCloud keys to unlock our content and store it unencrypted at rest.
Maybe they can also build an api so that governments can search easily for dissidents without the delays that the due process of law causes.
Facetious as this is, I can't imagine this is anything other than Apple's endgame here.
The best of both worlds: keep advertising their privacy chops to the masses, while also allowing any and every government agency a programmatic way to hash-verify the data passing through their systems in real-time.
The proposed attack on Apple's protocol doesn't work. The user's device adds randomness when generating an outer encryption key for the voucher. Even if an adversary obtains both the hash set and the blinding key, they're just in the same position as Apple—only able to decrypt if there's a hash match. The paper could do a better job explaining how the ECC blinding scheme works.