Scanning “private” content
lwn.netI haven't seen much discussion of the changes to search, which seem a little dystopian as well:
> Siri and Search are also being updated to intervene when users perform searches for queries related to CSAM. These interventions will explain to users that interest in this topic is harmful and problematic, and provide resources from partners to get help with this issue.
Will apple.cn be extending this to searches about "tank man" or a certain stuffed bear? Oh bother...
I noticed google doing this too while trying to look up case law related to the discussion of apple's spyware. Related google searches popped up a big intimidating notice: "WARNING Child sexual abuse imagery is illegal".
The obvious implication is that your searches are being reported to some unaccountable authority.
I have no doubt that this creates a chilling effect against public discussion about these practices.
Lol. I've gotten these many times. It turns out that an acronym related to my work is one typo away from a listed child porn term/site. If google is reporting these then I am already on thier list.
In particle physics, it is interesting which fundamental interactions conserve parity, charge parity and time reversion. Interactions that do not conserve one or multiple parties are called "parity violating". In this context, in some interactions combined charge and parity conservation is violated, which is unfortunately named CP violation.
Better not look up anything about /usr/bin/cp.
For the Hugin panarama suite one tool to find control points is named cpfind.
Civil Protections in Half-Life 2 is also often abbreviated CP. IIRC the are even CP violations mentioned in the Overwatch announcements, which would fit in with the series' physics references.
So many minefields...
Or look up how to handle kill process or sacrifice child.
> It turns out that an acronym related to my work is one typo away
Ah that old chestnut, I am sure they have heard many worse excuses than that one.
There is an old Drew Carry joke:
"I wanted to get the Playboy channel, but I already have the cartoon network and if you have both they put you on a special list."
The warning suggests that your search might return child sexual abuse imagery, which is odd.
If Google was doing its job, no search results would contain CP and they wouldn’t need to warn anyone. They probably shouldn’t warn anyone since all it does is alert people who might be looking for CP that they are being watched, and scares anyone who isn’t looking but their search terms have similar words.
The outcome is overall worse than if no alert popped up.
You used to get that when searching for PDO (PHP Data Objects) documentation on Google. I think it got fixed some time in 2019 though.
If you search anything related to eating disorders on Tumblr you’re presented with basically the same thing with links to help resources. MyFitnessPal pops up a similar message if you log less than 1000 calories in a day. Same with topics related to suicide on any search engine.
There is a very real bright line difference popping up a “please get help” message for self destructive behaviors and arbitrary censorship.
Source: have eating disorder and consumed thinspo and proana content in my teens.
This is interesting because it’s trivial to use someone else’s Siri and make it search whatever you want.
(I can trigger my partners with 75% reliability if I just speak in a high pitched voice)
Well, it would certainly be an interesting moment in broadcast if someone were to ask millions of Siri's everywhere to search for "the forbidden things."
then they literally have a recording of you saying it...
in a voice that doesn't sound like their voice.
on edit: changed your to their
> Child pornography and other types of sexual abuse of children are unquestionably heinous crimes; those who participate in them should be caught and severely punished.
Have you seen an article that, say, criticizes prison system and needs to start with a reminder than killing people is wrong? Definitely a strange sight. This looks like Soviet era prefaces about the decisions of latest CPSU congress and some relevant opinions of comrade Brezhnev that were expected to be found in any decently sized publication, whether it was a material science textbook, or a paper on Babylonians. It doesn't matter what you think, just do the required dance.
This might seem as nitpicking, but that preemptive display of obedience is the very thing that allows the likes of Apple and its customers to use the pretext successfully.
CSAM is just a very touchy subject (pun not intended) because people really loose their shit easily if you mismanage your words. It's kind of weird actually, no other crime can set people off so badly. And in the age of online witch hunting, well... I get why the author does it.
It is usually lazy to point at evolution in social situations, but I think "look after the babies" is probably an instinct that has been baked in to humans at a very deep level. It explains why the response is often positive but surprisingly strong and unreasonable (it is almost humourous the list of atrocities that are easier to defend than CSAM, and grim how the instinctual protections offered to children fade away when people become adults).
Children aren't very tough. If they don't have nearby humans intervening to protect them they tend to die or do badly. Evolution favours people who have strong instincts to protect and promote children. It is plausible.
Whether or not the instinct manifests to some degree in everyone, it truly goes into overdrive once you become a parent. It's hard to describe in all its scope. Even a concept of a child being hurt by someone starts to become extremely disturbing.
For example, I've noticed that ever since I became a father, I can't enjoy some of the movies and shows I've liked previously, because the scenes where children are threatened or implied to be hurt become emotionally overwhelming to watch. To use a light example, take Star Trek TNG: Power Play (5x15). It was one of the episodes I always found boring, but when I got to it during the last semi-regular TNG rewatch, I had to pause it and collect myself. Almost switched it off. All because of one scene, where a mother and a child end up in the middle of a hostage situation.
One can invent a biological explanation for anything, that's the problem. The “instincts” in that reasoning is a stereotype anyway.
Another problem is that “childhood” is pretty recent invention (there's enough popular and scholar literature on that). What you mean as a “child” is not what someone meant mere couple of hundreds years ago. And kids actually died all the time through the whole human history, whether “protected” or not.
> And kids actually died all the time through the whole human history, whether “protected” or not.
I think that would almost be a point in favour of the idea. The response to CSAM is, frankly, irrationally strong in the modern era. As with many human emotions it looks to me like something calibrated for a different time and altered circumstances.
It is a lazy theory though.
And not just that... you can commit many many crimes, or even get falsely accused, win at court, and noone cares.
Just one headline "bruce343434 was found with CP on their phone!" is there for life... child-stuff and rape are one of those two things, that never seem to vanish, and just a mere accusation destroys lives, even if later proved to be untrue.
That's what I'm talking about. The questions arise immediately: Why is it a very touchy subject? Who made it such, and how? Is it the same for people outside of globalized US-sensibility-centered bubble? Is it a cultural thing?
There is a widespread belief that proper technical solutions are enough: give us end-to-end encryption and such, and everything will be all right. But people do things for a reason, and technical solutions are introduced accordingly. We need to look there to understand what's happening.
For an outside observer, the messaging media filter example does not even look convincing. So there's a Bad Guy chatting with a kid who can organize their meeting or take the conversation to a different, non-filtered service. And that's completely okay! (Unless, of course, there happens to be a need to promote Big Brother processing all conversations to protect the kids, heh.) However, when we mention sexual content being sent or received, there's a sudden wild flight of fantasy and countless dangers on the horizon. It makes one wonder whether the real goal was to protect no the kids but the parents, from the worried thoughts that their kid is not completely isolated from sexual sphere. The outcome is that today Beavis can't tell Butthead “Wow, look at those tits!” and send the picture without being reported. What a perfect repressive Victorian childhood, and what an outstanding member of society it will produce!
The taboo is twofold. One the one hand, it creates new positions of power for the people who enforce it legally and in the discourse who won't just dismantle themselves (quite the contrary, see the worldwide practice of drug prohibition and its effects on laws and bureaucratic growth). On the other hand, it creates the inflammatory excitement about the topic in the common person. Media knows well which stories — told from the correct angle, obviously — attract public. As a result, there is a stereotypical image of a maniac hiding in the shadows, and the need to “do something about it”. In fact, maniacs (also a stereotype formed by media, by the way) are rare: in 8-9 out of ten cases of child sexual abuse it's a person close to them who decides to “search for happiness” together in such a way.
If the father makes his daughter send him sexual photos, and he is also the one who gets notified about it, what is the point? Observe the observers, too? That's a bureaucratic dead end. Or is such system, the one to silently look into too numerous parental approvals, already in place? Then what about the father who gets notified about his daughter sending her boyfriend sexual photos while being completely okay with it? Well, maybe such father would disable the feature to stop feeling like a third wheel, but what would be the opinion of Big Brother? Effectively, the point of view of “the whole society” that you could previously silently ignore is transparently codified and enacted by the computer.
> If the father makes his daughter send him sexual photos, and he is also the one who gets notified about it, what is the point?
> Then what about the father who gets notified about his daughter sending her boyfriend sexual photos while being completely okay with it?
Continuing on with your scenarios, what about an abusive father who gets notified his daughter is sending her boyfriend sexual photos or texts, gets jealous at the "competition", and then uses these anti-predation systems to further isolate his daughter? The phone that would have been a lifeline to the outside world then becomes another tool of control.
> Have you seen an article that, say, criticizes prison system and needs to start with a reminder than killing people is wrong?
I'm pretty sure I have, actually. In the age of Twitter mobs that can have you fired over a misplaced comma, nothing should be left to chance. And protecting yourself from a misunderstanding or ambiguity isn't enough anymore, as a lot of the time these people are malicious. They will not only twist your words, but make things up entirely in order to make you look bad. And when (if) you get your 30 seconds to defend yourself, you'll want to have something short and unambiguous right at the top of the page to point at.
A problem is everyone giving in to outrage. There are some negative side effects for everyone, but just ignoring it is the most viable PR reaction. The people don't want to forgive anyway, they want emotional acknowledgement that nobody can supply on the net.
We have books being scanned that were written 15 years ago that now have to be rewritten. Publishers should never acknowledge these criticisms, this is a form of the worst censorship literature has to content with.
Of course ignoring everything isn't a solution as well, outrage can indeed be justified. But I think outrage has to surpass a level that summons more actions than a twitter post or change.org petition. If outrage still persists, maybe there is a problem you need to address.
Companies that comply too willingly would also do it for any regime. It is nothing that should be praised at all, especially not seen as "progressive" or "tolerant". It should be an example about how the free enterprise indeed fails to act responsible and champion any values. That would be a healthy perspective.
I agree, but allow me to make a somewhat controversial analogy: this is the same as vaccination and herd immunity and seems to be something that people are instinctively opposed to.
Just like vaccination in a pandemic, ignoring the Twitter outrage is the safest thing when everybody does it (herd immunity). But currently, very few are doing it and since it is dangerous to the individual (re. vaccine analogy: while they ARE safe, many people BELIEVE them to be dangerous, so the result is the same), nobody wants the be the first. And since you can't know if others will follow your example, it's easy to feel like a guinea pig, so you give in and do the long-term worse thing instead.
But unlike with vaccination, the individual risk is very real and more importantly, you can't use moral arguments to convince the individual to take on personal risk to benefit the group like you can with vaccination (and even that rarely works). Companies, unlike people, are not expected to be moral actors (and I'd argue most have to be by definition immoral). So why would a company, journalist or anyone else that finds themselves the target of an outraged mob (or sees a real risk of that happening) not take all the necessary precautions to protect themselves?
Let me join you in your sorrow over thousands of years of easy living humans enjoyed before Twitter.
I think it's because it's very easy to just say: "wait, you're against Apple fighting CP? Are you a pedophile?". People DO think like that.
I really wish they would avoid stating this belief of theirs as a fact. I do not enjoy being told what I should think. Just get straight to the topic instead.
The Butlerian Jihad cannot come soon enough. As time goes on, I realize that certain authors had better insights into the human condition than others. For example, Alduous Huxley has turned out to be more prophetic than George Orwell. I believe Frank Herbert will as well. Man cannot be ruled by machine.
This cannot last much longer. A lot of political division in this country is due to technology. As someone genuinely interested in computation, it upsets me to realize this, but more and more it seems inevitable.
I think the mistake here is that maybe man can't be ruled by machine but men using machines to rule other men..
It is already game set match.
Endless wars, right and wrong language, always on connected devices monitoring us constantly? Orwell seems to have a point.
So what you are really saying here is that "there is no sanctuary"?
Which country?
How would you organise a rebellion on platforms controlled by the same entities you're rebelling against, ones who have shown that they can exert tremendous censoring power? They've also become an essential part of life for much of the population. As much as I'd love to be proven wrong, I don't think there will be any mass revolutions or other "liberty tree watering" going on in the near future. The masses have already been beaten into submission and thoroughly enslaved.
> How would you organise a rebellion on platforms controlled by the same entities you're rebelling against, ones who have shown that they can exert tremendous censoring power?
How were they historically? In person, where people gather.
> The masses have already been beaten into submission and thoroughly enslaved.
Is this simply because technology exists, or because it is monopolized? I think it's the latter, and it certainly projects a path out of this situation.
The masses have not. I visit my parents out in their conservative suburb fairly frequently and life cannot be more different than what's represented on Hacker News.
Or at my parish, which is becoming more and more like family.. people are fed up.
It's easy to organize as long as you can gather in real life at restaurants, pubs, parks, churches.. oh.
Yeah lets meet. oh, I'm on the other side of the atlantic. lets meet in the middle then?
While I'm over here, a schizophrenic terrorized by the government, and I cannot wait for an AI to replace people. I want AI to overthrow humanity and subjugate us with a simulation, except with German orthography instead of Kanji (thus implying it is a German AI, not a Japanese AI as is canon in the movies, because I'll take my chances this time around). I want Cortana with a moral compass that could slice angstroms, not some Japanese bumpkins and CIA salvia-dealers' cloned consciousness with excuses.
Can you imagine: everyone at the NSA is celebrating Apple and the CSAM automated scanning.
Simultaneously, connected intelligence officials: wait, they aren't going to let the nation state AI judge what I do for the STATE? Surely, no god-like AI would understand what is necessary. No xir.
You should be forbidden from calling it private if you scan it like this. Companies should be forced to market their data storage as public so as to not deliberately mislead users
Public might not be the right word either, but I totally agree that it shouldn't be allowed to call it private. Maybe we need a new word for "not public, but not really private [from your corporate/government overlords] either".
Is the fact that we need this new word a sign that we've gone too far? It's 1984 already, isn't it??
> Maybe we need a new word
Super-private, as in Supervised Private Data. :D
Imagine what it will be like when everyone has a neuralink. They will be able to scan for prohibited thoughts and stop you from thinking them before they're even fully formed in your mind. A world of infinite control and perfect morality
We naturally do this to ourselves: programming ourselves using positive and negative reinforcement/stimulus. Our “rational” selves are capable of training some odd responses into our “irrational” selves. Some people do it very purposefully.
One of the central themes of 1984 was thought-crime and how one was trained to recognise and avoid it (without ever specifying the parameters of its definition!)
Someone else's icloud photos got into mine once, just random sunsets from some place I'd never been. It was weird.
How can we trust this?
Maybe get the hash database and create a trove of clean documents that force a hash collision and overload the system with false positives. Maybe also add something about the Fourth Amendment in the documents for good measure.
> get the hash database
Apple designed the system so that their hashes are never known to client devices. Their server is fundamentally involved in checking your hashes against their list.
The article said they get the hashes from some government entity. Perhaps it is available from there.
You mean the NCMEC? They have the hashes, yes, but they won't just give them out.
They do, after signing serveral NDAs. There was an article here few days ago describing the process as a small hosting operator.
You are assuming people will not be marked for life for their false positives
I think the idea is “if everyone is marked for life, no one is”.
When everyone is on a list, it's the people not on a list that will be suspicious.
That's the point!
I don't know why people are so shocked about this. Apple has always been the shopping mall of personal computing with brand prestige coming ahead of all other considerations.
Librem 5 and pinephone are the future
What about Fairphone? EU only (or isn't it?) so I guess it's not a choice for that market where Apple's scanning has been introduced already as of today, but still. Though I've bought an iphone to get away from Google's all-seeing-eyes and won't crawl back to Google's ecosystem now, so I'd like to know more about Fairphone's O/S options. Is their Android de-Googled? AIU they're at least allowing installing alternative OSs without loosing warranty.
Fairphone doesn't offer a custom OS, it's just stock Android. There is nominal support for more open ROMs, but those are third-party (community) efforts. Their selling point is focused on the hardware, not the software.
(edit: you can buy a deGoogled (but still Android) Fairphone from eFoundation: https://esolutions.shop/shop/e-os-fairphone-3-plus-fr/)
Is Apple's approach even likely to catch any pedophiles? Seems to me at most it would succeed in keeping such photos from Apple devices, which is perhaps good for their public image, but does nothing to catch predators?
It is very public now that Apple will scan for such pictures, so how many pedophiles will keep them on their phones?
Or Apple is taken over by pervs who want to be able to look at private photos to "verify" them. Once algo reaches a threshold they'll be able to see "lowres" versions.
Apple will refuse requests to add other types of hashes …
Sure, but when it’s mandated by law, what then? How difficult would it be to force insurrection related imagery to that hash database? Apple would be in a very tough spot if that were made into law.
They’ve already opened a can of iWorms, because now that they’ve shown it can be done, lawmakers in any country can mandate the same technology be added to every device and it doesn’t have to be restricted to CSAM. That’s just the family-friendly excuse.
A small reminder that, iPhone in China holds their icloid data in China(雲上貴州) due to request of government. Will they ever reject request from government if that can harm their sell? I doubt.
Apple has no way to know what image hash has been derived from. So China can give a hash of Winnie the pooh and claim it is CSAM. Apple won't know.
Spot the Apple employee in the comments who wasn't the one who leaked the internal NCMEC memo..
Sending images of authority abusing its powers? No no, can't do that, back to gulag.
Does anyone know if this also scans files in your icloud synced desktop or documents folder?
It does not. Only photos (designated to be) uploaded to iCloud Photo Library.
iCloud is not P2P encrypted so it's likely that they already do. They can do it without your knowledge or having to push a software update.
This is something we almost certainly didn't raise any issues with regards to Google Drive as well
Different. Many people are okay if you upload files to a third party that they are then scanned by that company.
People here are not okay with a local scanner installed on your device. Despite Apple “promise” it will be only used for photos and those you “intend” to upload…. this falls on deaf ears. If you were only going to scan content intended for the cloud…. then just scan it on your servers when it arrives.
Despite talks of E2E … apple has not said this is the reason for the local scanner.
The scanning is not done in iCloud.
But does the location really matter? At this rate its cloud service providers up next.
Cloud providers already do scanning. Which is fine - it's their hardware, and their service terms.
That is vastly different to a device purchased by a consumer, deciding to do something that the consumer has no control over, which may cause irreparable harm to the consumer in the case of a bug or false positive.