A Calif. Teen Trusted ChatGPT for Drug Advice. He Died from an Overdose
longreads.comIt's a good read, actually. Includes actual specifics from the conversations, how the user phrased his requests, how ChatGPT responded, and why safety systems seemed to have failed (ChatGPT generally has a policy of not giving advice on illicit drug use, but that broke down here).
It is fairly straightforward to bypass the safety protocols just by slowly shifting the conversation towards the prohibited topic.