For a few nights in a row, I woke up to alerts about a huge spike in sign-ups. By the time I started troubleshooting, the activity had stopped. I activated CAPTCHA and moved on.
When the pattern repeated yet again, I decided to do a deep dive into the data.
The scheme
What I found was that the hackers were...
creating thousands of accounts
adding a valid payment method to each account
running a single very expensive LLM call (2-3 USD)
This would let the first request go through, then trigger a charge to their payment method. The payment method gets rejected, but the request has already been processed. Using this method, they would get away with about a thousand dollars' worth of credits every night, which kept them interested in the service.
The timing
But what caught my attention wasn't the money – it was the timing. The attacks coincided with my sleep cycle.
Coincidentally, that day I decided to take a break and disconnect from my computer early. Just 30 minutes after I shut down my computer, I got the first notification.
I logged in to check, and it stopped.
Went to play some games and ... 30 minutes later, I got the second notification.
They were checking my Discord status to see if I was online.
Sure enough, I confirmed this by setting myself as offline on Discord, and the attacks popped right back up.
The game
I didn't want to remove free credits for everyone, so I decided to mess with the hackers and use them as my personal pen testers.
The first thing I tried was requiring proof-of-work on sign-up. I shipped the change, went "offline," and watched the logs. Within an hour, the sign-up attempts resumed – they'd already integrated a solver. So I went back "online," and the attacks paused while I added JA4 fingerprinting. Back "offline." This time it took them most of the night to find a workaround. I'd watch them troubleshoot their automations until they figured out a way around it, then go back "online" and layer on the next thing.
I mostly forgot about the entire incident until they came back to try their luck again the following week. But despite a few nights of alerts about tripwires getting triggered, they never managed to get more than the few cents we deposited into new accounts – not enough of an incentive to keep trying.
In the end, it was the cat-and-mouse game that made the whole experience worth it. I got free pen testing; they got a few dollars.
Card testing vulnerability
I wasn't surprised about the overdraft feature being abused. This was something we were aware of and treated as a conscious trade-off between convenience and risk of abuse.
The bigger issue was that this made me realize that a malicious actor could abuse our system for card testing. That's a widespread problem and one that will get your Stripe account flagged. When researching this problem, I didn't find many effective solutions, so I wanted to dedicate part of this blog post to sharing what I learned.
Here's what I tried and how it held up:
Method | Effectiveness |
Device fingerprinting | Ineffective. Fingerprints are great for detecting legitimate returning users (e.g. to bypass CAPTCHA), but because they are easy to fake, they are not effective at detecting malicious actors. |
IP address blocking | Ineffective. Residential proxies are cheap and easy to get. |
CAPTCHA | Mild deterrent. Ineffective. Many existing solutions to bypass CAPTCHA. |
OTP | Mild deterrent. Ineffective. Many existing solutions to bypass OTP. |
JA4 | Somewhat effective. JA4 is a TLS fingerprinting method that identifies clients based on how they negotiate TLS connections. Of all data points that we collect, JA4 is the most stable identifier. |
ALTCHA | Somewhat effective. ALTCHA is a proof-of-work challenge that requires the client to solve a computational puzzle before submitting a request. When combined with prior methods, can slow down the attacks enough to deter the attacker. |
Rate limiting | Somewhat effective. Slows down the attacks, but may hurt legitimate users. |
At the end of the day, each method is individually bypassable – the game is making the combination expensive enough that the attacker moves on.
Oh, and set your Discord status to offline.