Downtime postmortem, wikis remain down for now, fuck AI
Admin CavemanJoe – 2026-03-16 19:20:45 [Permalink]
We started having intermittent downtime late on Thursday night, which I found out about because people were emailing me and pinging me on Mastodon (please bookmark our Mastodon page because when the Island goes down it's the only way we can tell you why it's down and for how long, https://mastodon.social/@improbableisland). I checked it out and all of our CPU cores were pegged to 100%, I could barely even get a hold of the server to ask it hey what the fuck.
Tailing some log files I saw thousands of requests per second going to our wiki, all from slopbots looking for English-shaped text to ingest. Anthropic, Google, Facebook, OpenAI, all the usual rogue's gallery of dickheads trying to build the next evolution of "How the fuck do I turn this shit off" before the money runs out.
What "tailing a log file" means is sitting at your computer watching fixed-width font scroll jerkily up a screen. Two monitors full of that. Proper looking like something out of a bad film about hackers. Each line would have an IP address of who was visiting, the address of the page that was being accessed, and the user agent string, which tells you technical details about the browser visiting the page. They'll say something like "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:148.0) Gecko/20100101 Firefox/148.0" or some shit like that, except when it's a bot, and boy oh boy was I seeing a lot of bot traffic, sat there in the dark in my Sinister Computer Hoodie.
Up until a couple of years ago, the bots were usually search engines, and they're supposed to first check for a file called robots.txt on your web server and follow any instructions they find there about what pages have useful information on them and which pages they should just ignore (it's not worth a search engine checking out the contents of your login page for example). The bots today are mostly AI scraper bots - they look at every page on your website and hoover up all the text so that they can figure out which words tend to go in which order. So, just write the names of those bots in our robots.txt saying "Don't look at this site, there's nothing for you there" and we're good, right?
Nope! Robots.txt is a request to not have your site scraped, and the people involved in AI are famously not concerned with consent. My robots.txt asked politely for the AI slopbots to ignore the wiki, and the slopbots went "lol no" and continued hammering the wiki so hard that the server couldn't keep up with them.
(Improbable Island runs on a dedicated server, a whole-ass computer just for one website, and that's vanishingly rare these days. Most websites are served from machines hosting dozens to thousands of other websites all from the same physical computer. And the slopbots still managed to hit it with so much gusto that it got so busy serving their requests that the actual humans using the site had to wait. You'd think that once a bot sees that its requests are taking longer, it would make fewer requests and give the site time to catch up - but the people who programmed these things are not smart)
So, the next step up from a polite request is a "No, fuck you, you're not allowed, get out," and that's Apache's .htaccess file. You can configure this to outright refuse to serve requests from anything whose user-agent string matches a set of templates, so anything that announces itself as a bot just gets an error message.
That worked really well. For one day. Then the bots returned, except this time pretending to not be bots - they'd changed their user agent string to look like humans. Except the damn things were dumb enough to still jump from page to page faster than any human could skim, and follow every single link on every single page. They continued acting like bots but wearing little hats that said "I'm totally a human" and they kept taking down the damn site.
So, I took the wiki down temporarily while figuring out what to do about this whole problem. The bots continued (and still continue) to knock on the door, but they get nothing but a little "Fuck AI" message with no links to follow. As I type this I've got the log tailing on my other monitor and they're still trying, a few every second.
Meanwhile, our Distraction and email-confirmation emails started bouncing, and I got a heads-up email from our datacentre saying yo, this server's IP address got listed on Spamhaus, sort it out or we'll cancel your contract. Uh-oh.
See, some of these bots had stumbled across the wiki's registration page, seen a form that looks like it takes an email address, shat out something that it thought looked like an email address and fed it to the form. The wiki then fired off an email to the email-shaped hallucination going "Right, if you're a real human, here's how and where to log in." (this was enough to deter spambots, but AI slopbots are even fucking dumber than spambots) Most of these emails bounced but some of them coincidentally happened to be actual email addresses. And of course these were being hallucinated a couple hundred times a second, so our email server was just sending out fucking thousands of emails, and the spam detector guys were like "Looks like spam" and they were right it bloody was, and the likes of Microsoft were going "Yo, according to Spamhaus you're a spammy spammer, so I'm not gonna deliver this email intended for one of your players telling them about a Distraction they just got from another player, sort your shit out." So that's why some of you weren't getting Distraction emails the past few days.
This all happened so, so fast, you guys. Wednesday night the slopbot attack started and I was up until 2am figuring out that it was in fact a slopbot attack (it initially looked like a hardware failure), by Thursday morning we were already marked as spammers, I was babysitting the server and setting up botblockers all of Thursday during a tornado warning, on Friday I'd gotten the slopbots blocked-for-good and the situation under control and my datacentre contacted and given the heads-up of what was going on just in time for the power at my house to go out for the remainder of the night, and we were marked as not-actually-spammers-after-all on the Saturday.
It has been, in a word, exhausting. It reminded me of being in my 20's and having to drop everything and get to a computer so I could figure out why the Island was down again (our code was way less efficient back then) and having it turn out that we'd been featured on StumbleUpon or linked to from a massive gaming blog or whatever and suddenly had two thousand newbies running around the place - except instead of a sea of new faces, it was just slop. Enormous amounts of kilowatts of useless, pointless fucking slop. And I'm not in my 20's anymore, I'm in my 40's, I've got a kid, I can't just drop everything at a moment's notice anymore, that's a young man's game. First time in a long time that I'd genuinely considered packing it all in. But, y'know, you figure it out and get over it and carry on.
Our next steps are to set up the wiki so that instead of making a separate login, you'll use your current Improbable Island credentials. That gets complicated for people running multiple characters and it's the reason we haven't done this before now, we've had a one-account-multiple-characters system in the works for a while and I was waiting until that was ready, but we can't really wait on that anymore. Once we've got that system up and running, the wiki will return, but you'll only be able to access it if you've got an Improbable Island character who's done something human-looking in the game.
I am sorry about all of this, it feels bad to put those sorts of restrictions on people just to fight off bots (and also to waste my time with this shit when it'd be better spent making y'all nice new features), but until the tech industry and energy companies move on to the next scam we'll just have to deal with it and get creative. If it takes longer than a few more months for the AI bubble to pop then we'll set up some kind of low-resource high-information-density maze for the bots, or start feeding them their own shit to try to help move things along. Fortunately (?) we're not alone in this, an increasing number of website owners are as pissed off with it as I am and there's some prior art (this blogpost about fighting back against slop crawlers goes into some detail (if you're using Brave you'll also get the iocaine, so use a different browser)).
Anyway, there's your postmortem, I'll have the wiki back up as soon as possible and then we can move on to more interesting things.
Have fun,
~CMJ
Intermittent downtime
Admin CavemanJoe – 2026-03-12 14:40:19 [Permalink]
Hey folks, looks like we're having a bumpy time of things. Please check https://mastodon.social/@ifixcoinops for status updates.
Update 2026-03-12, 14:30 server time: it was fucking AI scraper bots trying to ingest the wiki. I've put some protections in place - the well-behaved bots will follow them and quit bugging us, now to figure out what to do about the not-well-behaved ones.
AI is a fucking scourge and I cannot wait for the bubble to pop.
Cheers,
~CMJ
Start your engines!
Admin CavemanJoe – 2026-03-10 14:38:49 [Permalink]
Hey folks, the Monthly Memento for March is the Marching Boots, which gives a small but permanent Stamina cost reduction to travel across plains. We needed a travel-based MM because I've finally released the new Rally system!
The new Rallies replace the fun-but-too-long old Hundred-Point Rally feature up in Cyber City 404, where you'd race around the map to win req and cigs. The new version is way better, with shorter and more frequent rallies, and you can compete in multiple rallies simultaneously if you like.
Head to a Rally Headquarters in any Outpost and you'll see a list of current, future and recent rallies. Each rally has a number of map points to hit, either in a particular sequence or whenever you come across them (most rallies are a mix of both). The first player to hit all the points gets half of the prize pot, the runner-up gets a quarter, and the remaining quarter is shared between everyone else who finishes the rally before time runs out. Payouts are rounded up, so it's ofter possible to get a cigarette or two for finishing a rally even if you come last!
Rallies can be a chill way to wander around the map checking out cool Places, or a frantic sprint to get the fastest time, play them however you fancy and have fun!
~CMJ
Holy shit it's January (EDIT: Girl scout cookies!)
Admin CavemanJoe – 2026-01-19 17:50:53 [Permalink]
Hey folks, January's Monthly Memento is obviously the Shiny New 2026 Calendar - that's an extra game day per real-world week, just like we do every year when I can't figure out what to do for January.
What have I been up to lately? I've been up to my elbows in Rally code (see previous MotD) but also up to my knees in General House Shit because yup we live in a hundred-year-old house and shit gets fucked sometimes, but while I've been in the Rally code I keep getting pissed off with silly little things and going "You know what, I deserve better," and deciding to save myself some time later on.
Lemme give you an example, currency display.
Anywhere in the game where I have to display a cost of something, or you getting something, I've gotta write code that goes, like, are we dealing in cigs? Yeah? Well, are we talking about one cig, or more than one, because if the game puts out "Buy for 1 Cigarettes" then it's gonna look like amateur hour in here, also shall we write this code twice because the mobile buttons are small so they say "Buy (1 cig)" and they'd stretch out or look weird on small phones if they said "Buy for 1 cigarette," also shall we use those nice new icons that I made during covid lockdown? And of course let's make sure that the player has enough Req and cigs, and wouldn't it be nice and handy if rather than saying "You can't afford it" it'd say, like, "You're 28 Req short." Obviously if the player's on mobile we wanna pop up those handy little icon-based thingies in the bottom-right corner that show how much currency they have, to save them scrolling down, and when they buy the thing we want to put that in the debug log too, so let's just write a quick thingy for that... Oh that's right I was trying to use a standard colour for req and cig costs too, now what was it again...
So yeah, there's a lot of bollocking around literally every time the game asks for or gives currency of any kind, and after eighteen years I finally got pissed off enough to write myself a currency handling lib file so I can literally just go, like, displayCurrency($req,$cigs) and it'll show the req and cigs cost in a standardized way and take care of doing the mobile icon thing, or I can go cannotAffordMsg($req,$cigs) and it'll go "You're short by 1,862 Req and 6 Cigs." It's all code that I only have to write once and then I can use it wherever and honestly I feel really daft for not doing it before now.
Oh also, time-related functions. I wanna show the various Rally starting times in your local timezone 'cause although it's nice and part-of-an-international-community-feeling to give all the times in GMT/UTC, this ain't the BBC World Service, this is absolutely not Srs Bizness, folk are just tryna play a fun text adventure here, let's make this bit a little bit easier. Problem: the game doesn't know your timezone. I don't want the game to know your timezone, because that's personally-identifying information and I treat that shit like radioactive toxic waste and want it as far away from my server as possible, it's legit a fucking liability, every little bit of dirt we have on you there's pain-in-the-arse procedures for, never type your real name into my website etc etc check our privacy policy keep your damn mouth shut etc etc. So, we gotta come up with a convoluted-ass way of doing, like, the server says to your web browser "I'm sending you a number, it's the number of seconds since midnight on the first of January 1970 at the Royal Observatory in Greenwich plus 3,360 seconds, figure out the rest yourself and don't ever, ever tell me about it you snitchy little rat," and the browser goes "Cool, I know what time it is here on this phone and I will roll around in javascript like a pig in shit in order to transmogrify that number into a sentence that says oh this rally starts in fifty-six minutes at 4pm and I'll just quietly change the number right here on this little screen and never tell anyone about it," and the server goes "if you just said something I wasn't listening."
This is a little bit different to how most websites handle time offsets. Old-school forums used to ask for your timezone when you joined up, but more modern websites just persuade your computer to snitch it to them and store your timezone offset on their server. I don't wanna do any of that for reasons of, well, look around, in the present climate it's probably inadvisable to store any data about anybody at all, and to answer "What are you, a cop? Fuck off" to strangers asking for the time. Convoluted way of saying now I've got a lib file that converts server time to local time but there you go.
Anyway, I hope to have the new Rally system in place and operating in early to mid February, so watch this space, and have fun!
EDIT: Players in the USA can buy Girl Scout cookies from my kid so she can get a plush ferret!
~CMJ
Holy shit it's December
Admin CavemanJoe – 2025-12-09 19:03:29 [Permalink]
Well heck, I never even announced the November Monthly Memento and it's gone, we're into December now, it's Solstice Squirrel season again!
Currently I'm working on a replacement for the Hundred-Point Rally, and you might've caught me chatting about it in Banter, but for those who haven't: the Hundred Point Rally is unreliable, too long, and not fun enough, so I'm writing a whole-ass new Rally system to replace it. Each Outpost will have its own Rally HQ, each species will have its own personality for how it goes about rallies, and the general vibe is "Shorter, more interesting rallies, some fast, some chill, you can run multiple rallies at the same time and everyone who finishes a rally will get some req/cigs even if they're in last place."
I don't wanna give a timeline 'cause you know how timelines go, but I'm getting a solid few hours a day in on this and hope to have it out for you Pretty Soon.
Have fun!
~CMJ
October's Monthly Memento
Admin CavemanJoe – 2025-10-23 17:07:26 [Permalink]
Hey folks, October's Monthly Memento is the Skeleton Mask, and it can give you a minor all-day reduction to your monster encounter rate when travelling on the map.
Just little bugfixes this month, been up to my arse in vet bills and car bullshit. I did restock the Gargantuan Candy Bombs though.
Have fun!
~CMJ
September Monthly Memento, downtime
Admin CavemanJoe – 2025-09-19 20:55:04 [Permalink]
Hey all, we had a day of downtime due to a dead drive and a series of fuckups following the dead drive. Should all be back to normal now but please give the place a good poke, go do things that you don't normally do, check out the minigames you forgot we had, all that fun stuff while I keep an eye on the logs.
September's Monthly Memento is a pipe, because I spent weeks burying two hundred feet of 4" corrugated pipe in my garden to try to stop my basement from flooding. You can swing it round your head for a buff, or smack a monster with it for a different kinda buff, it's a twofer, sometimes the monthly mementos are fun like that. Hit that donate link and gimme a tenner to get it.
Stay tuned for bugfixes and maybe a couple new little things here and there, and have fun!
~CMJ
August Monthly Memento
Admin CavemanJoe – 2025-08-19 03:32:03 [Permalink]
Holy shit I never announced August's Monthly Memento, well it's the VPN Profit Share certificate, in honour of some dumb shit going down in the UK. This Monthly Memento just straight-up gives you money, a nice 50 Req bonus each game day. Not much, but a nice leg-up in the earlier levels.
I'll have some more neat new game features for you later in the month!
Cheers,
~CMJ
July Monthly Memento, drive labs bugfixes
Admin CavemanJoe – 2025-07-07 16:11:47 [Permalink]
Hey folks, July's Monthly Memento is the Pride Extender, because the big shops round here were too chickenshit to do pride merch this year and apparently it's up to us to pick up the slack. It's a weapon that you can equip at the start of a DK run to get a little advantage without having to visit Sheila's.
The Drive Hunt mechanic has been a bit busted the past couple of weeks; normally labs will move around the map but I had temporarily disabled that feature because of a rare bug that could occasionally make two labs appear on the same square. I've fixed it now, so labs will move around correctly again.
In case I haven't mentioned this before, how the drive hunt mechanic works: there's a set of lab locations that applies to everyone, and a set that applies only to you. Everybody starts a DK run with the same starting lab locations. Once for every game day that you trigger, there's a small chance for each lab to move - but it only moves for you, not for everybody. Once per Midnight, there's a very small chance of the starting locations changing (IE they change for everybody, not just for you) - but, if you've observed a lab, either by chucking a dart at the Comms Tent or by coming across it on the map, then its position is locked (for you specifically) and it can't move again until your next DK.
When a lab moves, it's not random, there's some logic to it; a lab will only move to jungly or mountainous regions, and it'll try to avoid being too close to other labs or outposts.
So there's value in asking other players where their labs are - but not much value in asking, like, one or two people, you need to really ask around to get a good idea of where's most likely. Furthermore, if you want to know lab starting positions, finding them on the map early on in a DK run will give you a more accurate picture.
TL;DR: the more game days you accumulate during a DK run, the more your personal lab locations deviate from everyone else's. Keeping tabs on the lab locations in order to help each other out is a thing that I knew y'all would do, and I designed the system so that you could do that and there'd still be some "game meat" in it. Telling people where the labs are isn't cheating, it's part of the game!
Have fun!
~CMJ
Pride of Kittens
Admin CavemanJoe – 2025-06-12 03:36:49 [Permalink]
Hey folks, I'm late announcing it but June's Monthly Memento is the rainbow-coloured bundle of wires that I have to tidy up because these two new kittens keep trying to eat them omfg.
Say hello to Mab (black) and Herne (tabby), the new apprentices to Improbable Island's Official Cat Nola. They're living upstairs in Improbable Island HQ while Nola sniffs under the door so they can get used to each other. Once introduced, Nola will begin grooming them for Officialdom.
Financial contributions towards their upkeep and education can be sent to the usual address.
Have fun!
~CMJ