I’ve long believed it’s irrational to train people about scams. The two primary reasons are that I don’t think there’s any daylight between credulity and skepticism (either you walk into the propeller on account of an attack’s urgency, emotional appeal, legitimacy) or you’re skeptical and you don’t succumb. The second reason is that I don’t believe you can be taught, classroom style, about something you don’t care about (and unless you work in security explicitly, you don’t care much about this particular risk).
My financial institutions diligently fire off these emails but cui bono? They do. They’re building a case for denying severe victims reimbursement (ie, limiting their liability). And maybe they hope they can reduce fraud rates on the margin.
My long held conviction is that quaint ‘pause and verify’ advice helps nobody. Either I’m not a clicker or I’m a clicker (and there’s research to back this view up here and here.)
I once had a conversation with the brilliant CISO of AlphaSense and he referred me to the distinction between episodic and semantic memory. He argued, and I immediately agreed, that scams education is only compatible with episodic memory (a conscious recollection of specific experiences and events); semantic memory, the stuff of classrooms and textbooks, is the general store of factual knowledge without any personal context.
At Charlemagne Labs, we research frontier AI capability in social engineering and built the benchmark that Meta Superintelligence Labs used to assess a risk of its Muse Spark model. Our focus is on building Charley, a security assistant that defends employees against risks. Our northstar is to eliminate security awareness training in favor of continuously personalized education and realtime interventions. We do not intend to be a B2C startup so it’s worth mentioning that the AI-powered tactics of consumer social engineering attacks overlap significantly with those of enterprise victims. Business banks do the same thing as consumer banks in terms of hammering away at education.
Recently, my rigid belief about the futility of scam education has been destabilized
Last week I read with my own incredulity the news about ChatGPT being used assiduously by individuals to scam check. They don’t provide data on people qua employees, but it doesn’t matter because I was gobsmacked by the topline datapoint: more than 130M times per weeek!
So perhaps I am wrong, very wrong. Maybe awareness campaigns about scams, notably Craig Newmark’s Take9 campaign, are having an impact and creating a wedge where people don’t quite know what to believe and they seek an authoritative check.
It’s also possible that the crazy rise in prevalence of scams means that more people’s guards are up and then a virtuous cycle of social mimicry raises more guards. People love gossiping about scam stories and in a village sense, it’s depressing but delicious to talk about them. This doesn’t mean that official education is working, but perhaps that some kind of bridge between episodic and semantic memory is being built. I remember my friend or neighbor fell for a romance scam when I’m at the top of an attacker’s funnel and I pause to verify.
A third possibility is that OpenAI’s user base is so large and their engagement is so intense and high frequency that 130M times per week is a small fraction of overall chats; and the absolutely number of people scam checking could be static or even declining, despite the overall data point being so large. More transparency and baselines will settle this.
When I pitch Charley I often start with a discussion of asymmetry of warfare and the example of identical versus fraternal twins. You can teach people that fraternal twins are different heights or hair colors, but when you encounter identical twins there’s no point. The same is true of tomorrow’s social engineering attacks. Setting aside synthetic voice and video, if we look merely at text-based attacks the tells we rely on are disappearing. We simply leave too large a public footprint about ourselves for anything but highly verisimilar communication to be used by attackers and threat actors going forward.
Think your digital footprint is under control? A recent theoretical development exposes how our online participation puts us at risk of impersonation and deception in unfathomably arcane ways. Researchers have come up with Homophilous Network Learning (HNL). It’s like birds of a feather, flock together, but supercharged. Even a restaurant or bar review could be leveraged by an attacker to construct your network: “friends tend to write reviews of similar lengths, and that behavioral synchronicity leaves a statistical fingerprint across thousands of public posts.” If we’re online, we are on a battlefield against deception every day, both as individuals and employees with network access; our adversary is leveraging AI in ways we may only discover after the threat patterns are established and significant financial, brand, and business value has been destroyed. Until we have commensurate defense, we’re losing the war.