Before grad school I worked at a trust and safety consultancy based in Europe, auditing platforms for compliance with laws like the Digital Services Act. The job taught me to read privacy policies the way a lawyer does, which mostly means scanning for the one sentence that quietly gives everything away. And I still tell Claude things I would not say out loud. I know exactly what I am handing over, and I do it anyway.
And then a court made the stakes literal. In the copyright case The New York Times brought against OpenAI, Magistrate Judge Ona Wang ordered the company in May 2025 to preserve and segregate chat logs it would otherwise have deleted, even the ones users had asked it to erase, and to keep doing so going forward. Then, in January 2026, District Judge Sidney Stein affirmed a separate order compelling OpenAI to hand over 20 million de-identified conversations, a sample drawn from years of ordinary logs, to the plaintiffs. Two different orders, one quiet lesson: the chats you thought were yours to delete are the company’s to keep, and a court’s to call for.
The delete button, it turns out, was a request, and a court can override it.
Once you know that, the natural next question is what happens to everything you did not delete, and here the honest answer is that there is no single answer. There is no unified standard that every frontier model has to clear before it goes in front of the public, and there is no unified rule for what the company does with what you type into it afterward. It is set per company, in terms that change, and the two biggest products do not even agree with each other.
ChatGPT trains on your conversations by default. You can turn it off, but you have to know the setting exists and go find it, and your chats are stored until you delete them. Anthropic, which makes Claude, changed its consumer terms in 2025 and gave existing users until September 28 to choose. If you opt in to training, your conversations can stay in the pipeline in de-identified form for up to five years. If you opt in to training, your conversations can stay in the pipeline in a de-identified form for up to five years, which is a sixty-fold jump from the thirty-day window that used to be the default. Two products that hundreds of millions of people treat as interchangeable, two completely different deals, and almost none of it is legible to the person in the middle of typing.
What makes this different from a Google search is not really the storage. Google keeps your queries too. It is what we are willing to put in the box. A search is a few self-conscious words you would be a little embarrassed to have read aloud. A chatbot session is a paragraph, written in full sentences, about your health or your relationship or the numbers your startup has not announced yet, because the format invites it and the thing answers like it cares. The research backs up the instinct: The research backs up the instinct: in a KFF tracking poll, about a third of adults said they’d turned to AI for health advice and one in six for mental health, many because it felt more private than asking a person. People disclose more to the machine, on the stigmatized subjects, for the same reason they trust it less.
The part that should bother us is that the protection has not caught up to the disclosure, and the people running these companies know it. Sam Altman said as much last summer, on of all places a podcast:
“People talk about the most personal stuff in their lives to ChatGPT. People use it, young people especially, as a therapist, a life coach. And right now, if you talk to a therapist or a lawyer or a doctor about those problems, there’s legal privilege for it. We haven’t figured that out yet for when you talk to ChatGPT.”
Sam Altman, on This Past Weekend w/ Theo Von, reported by TechCrunch, July 25, 2025
He is right, and it is a strange thing to hear the CEO of the company say. Your conversation with a therapist or a lawyer carries privilege, which is a legal wall that holds up even in court. Your conversation with the product his company built carries none, which is exactly why a judge could order twenty million of them produced. The intimacy is brand new and the legal category for it does not exist, so it defaults to the weakest possible setting: ordinary business records, subject to subpoena, retained on a schedule you do not control.
I am not trying to talk anyone out of using these tools. I use them constantly, including to think through this essay. The point is smaller and more practical than a moral panic. We adopted a technology that feels like a private confessional and is built like a corporate database, and we did it faster than anyone wrote the rules for it. When I was auditing platforms, the gap I kept running into was between what a system was certified to be doing and what was actually happening to the people using it. This is the same gap, moved into the place we now go to ask the questions we would not ask out loud.
The version of this I can defend is the one you can act on tonight. Find the training toggle in whatever you use and decide on purpose instead of by default. Know that deleting a chat clears it from your view, not necessarily from the company’s hold. And use the oldest rule there is for anything that goes into someone else’s system, which is to not type the thing you would not put in an email to a stranger. None of that is satisfying, and none of it is a substitute for the legal category we do not have yet. How we end up protecting what people say to these things, or whether we decide to at all, is still very much being worked out, mostly by the same companies that benefit from leaving it unsettled.
Sources
Preservation order, Magistrate Judge Ona T. Wang, May 13, 2025 — primary
Full docket: NYT Co. v. Microsoft Corp., No. 1:23-cv-11195 (S.D.N.Y.) (CourtListener)
OpenAI must turn over 20 million ChatGPT logs, judge affirms (Bloomberg Law, coverage)
Sam Altman on This Past Weekend w/ Theo Von, #599, July 23, 2025 — full episode (primary)
ChatGPT for mental health support: a systematic scoping review (Sage)