For the server side component, the challenge is knowing the target of adversarial generation since, again, the perceptual hash used is secret. At that point, you are reduced to the threat present in every other existing CSAM detection system.
Yes, although I'm sure a sufficiently motivated attacker can obtain some CSAM that they are reasonably sure is present in the database, and generate the NeuralHash themselves.
> At that point, you are reduced to the threat present in every other existing CSAM detection system.
A difference could be that server-side CSAM detection will verify the entire image, and not just the image derivative, before notifying the authorities.
> Yes, although I'm sure a sufficiently motivated attacker can obtain some CSAM that they are reasonably sure is present in the database, and generate the NeuralHash themselves
Remind us what the attack is here? The neural hash and the visual derivative both have to match for an image to trigger detection.
I believe something like: an adversary creates an innocuous photo where:
* The photo itself is benign.
* The photo’s NeuralHash matches known CSAM.
* The photo’s image derivative is not benign. It looks visually like CSAM.
* The photo’s image derivative matches known CSAM per a private perceptual hash.
The above, combined, could have a victim reported to NCMEC without being aware they were targeted. Since Apple operates on image derivatives, they could be fooled unlike other cloud providers. That is the claim.
At that point, the victim could point law enforcement to the original CloudKit asset (safety vouchers include a reference to the associate asset) and clear their name. However, involving law enforcement can always be traumatic.
CSAM in the database is confidential, so the human reviewer just needs to be plausibly confident that they have CSAM in front of them. However, it’s not clear to me that you can pull off all three simultaneously. Furthermore, the attack doesn’t specify how they would adversarially generate an image derivative for a secret perceptual hash that they can’t run gradient descent on.
If someone wanted to plant CSAM and had control of an iCloud account, it seems far easier to send some emails with those images since iCloud Mail is actively scanned and nobody checks their iCloud Mail account, especially not the sent folder.
The question is whether the visual derivatives are checked against derivatives from the database or just against abstract criteria. That seems to be an unknown.
> However, it’s not clear to me that you can pull off all three simultaneously
Agreed. People here seem to keep assuming that you can, but so far nobody has demonstrated that it is possible.
