a giant rat with wings
Last night I got a threshold notification from anthropic - I run a few projects and so it didn't seem like an order of magnitude off what I'd expect (but certainly more). It was Sunday and I was stoned so added a todo to check on it tomorrow.
This AM I logged in and saw a surprising graph:
I toggled by workspace and found out it was all coming from townscryer.com, which was VERY surprising because out of all my projects this one isn't particularly token-heavy (at least given the current userbase).
tldr; Townscryer is an AI game-master/storyteller. You say what happens, and the story unfolds, etc.
I went to the analytics and the new user count didn't correlate with this usage pattern (the vast majority of people try for free and move on).
(^ only a few users in the last days when the tokens were ramping)
THEN I noticed there was one session a user had with 619 submissions (that's 10-50x a standard session!). I had never seen a number so high before in a single session.
Digging in further, a constant stream of submissions over the course of 3 days as the story unfolded (was this a bot?) Regardless, immediately I saw they had run out of tokens a LONG time ago (uh oh).
Turns out there was a bug where the app wasn't actually enforcing token usage :melting_face_emoji: Automated tests were supposed to cover this but weren't somehow.
A quick fix but I was SO thankful I had optimized the system to rely more on haiku than sonnet else this would have been a multi-thousand $ mistake 😬
Townscryer tracks world state as it/you generate story elements, which meant a LOT of data to sift through to figure out if this activity was legit - a job chatgpt could probably solve?
Yep! (i lolled at human messiness)
So it turns out it WAS a real person and despite my mistake, the system performed far better in this stress test than I expected! The lesson? write tests? don't be dumb? maybe put in financial guardrails for token usage? idk but an interesting twist for my Monday.