Serious Sam handled massive amounts of enemies on 56k modem connections
staniks.github.ioI was one the developers responsible for implementing the netcode on Serious Sam. We often slept under the desks in the offices at croteam after lurking usenet. One post in particular described the QuakeWorld prediction system which inspired us. That night we coded a simplistic mvp as a colleague (hi dan) tested it over an old 486 nix machine acting as a router that we could simulate lag with. This was well before the actual game was built around it
Why did you make the guys with exploding heads who run at you and their scream gets louder as they approach? I can still hear them.
Why wouldn't they? It's called great sound design. Games of the 90s and early 2000s put a lot of effort in that. See Thief the dark project. Sound design can bring more immersiveness than graphics alone. It's something forgotten in many games of today that keep beating the ray traced graphics horse
I think it was a rhetorical question. I wasn't a big fan of the SS series, but those damn kamikaze still echo in my mind. I've even had nightmares about them...
the way they start in the distance and blend into the rest of the sounds until... ".. wait what's that sound?"
I love Thief - I am playing new levels (I think its black parade)
One of my favourite games.
I think it was a playful rhetorical question.
Oh you can thank Creative for buying Aureal and throwing it in the trash. That crippled immersive/3D game sound design for a long time, I wouldn't be surprised if that had ripple effects l the way into today. Also there is no need to rag on ray tracing, a nicely ray traced game look wonderful. Ideally we have a feast for eyes as well as ears though.
The Witch in Left 4 Dead. Whoever cooked up the audio for that deserves an Oscar.
That's like one of the key selling features of the game. Great experience that also scars you for life.
Exactly. See also: first headcrab fight with only a crowbar in Half-life.
Huh, it didn't do anything for me, I dodged it on the first jump and caught it mid-flight baseball bat style. The pack of sonic dogs was way worse. And the beaked hydra, with the constant metal drumming, ugh.
This is THE one thing I remember from playing Half-life :)
The other thing is when it comes over the intercom: Forget about Freeman! Anyone still down there is on there own!
The worldbuilding in Half-life was so good. You weren't a character walking into rooms and enemies would attack you, like Doom and Quake. There was a big event happening, and you were just a part of it.
It's quite memorable in VR as well! If you get the chance I highly recommend playing HL Alyx with a valve index setup.
It was on par with my first playthrough of the original HL so many years ago.
The noises... the noises!
Which are actually voice lines of people in torment played in reverse.
I love how there's an article about how some legendary game was made, and someone in the comments casually goes "oh yeah, I built that, fun times". It's great.
Hacker News is so special for that. I posted a question about Django a while back and sure enough, in rolled the framework's creator to answer it.
That's great! Which creator was it? Simon Willison is very active here, I don't know if Adrian Holovaty ever was.
I once commented something to the effect of “that must have sucked” on a story about debugging a weird error on crash bandicoot and in comes the developer to tell me “yes, it did suck.”
I work at a big company and commented once that some decision was stupid and one of the top two engineers at the company dropped in to tell me I was wrong. I felt so honored. (And he's wrong.)
Hahah oh man, do you have a link? I'd love to have an opinion on who's right.
Internal company discussion boards... /sigh
The gist of it is...
The person who wrote v2 of a thing got a lot of shit, because there were a lot of "cons" that could have been predicted and mitigated.
I complained about them getting shit. Making a new thing is hard, and yes, there were "cons" but there are an enormous number of "pros," and you can't always block progress trying to enumerate and mitigate all of the "cons," especially if the "cons" are social in nature (which makes them hard to predict!)
I believe the company should have let the guy build v2 and see if it works. Let him test it in a few cases. Then he can try to shift to managing roll-out, and etc. Unfortunately, v2 became hugely popular instantly (wow, a problem I wish we all had!) and then a bunch of stuff went wrong, because the roll-out itself "should have been managed better."
So, #2 engineer-in-the-company came in to my comment thread, and documented the cons. Actually linked to a slideshow which showed each con on a separate slide.
Sure.
Here's the problem, v3 was written by a TEAM of people, with backing from leadership, and the "cons" of that job are enormous and embarrassing. I mean, really bad. Years later. Unpopular, and most people haven't migrated to it.
Stop giving that one guy shit when a team of people did worse. And especially if your main problem with his work was that it was too successful too fast!
Grrrr.
Oof, that sounds like some dysfunction. Can't really argue with adoption, though.
Pity this was in internal forums, I thought it was on HN.
HN is wonderful for that. I once commented on an article about Brave Browser that it was an ad extortion racket, and Brendan Eich showed up to call me an asshole. Good times!
I never called you an asshole, I quipped that your handle's surname suits you. Accuracy matters, especially given that you are the source of the ass* name.
Hello, Brendan!
I don't personally see much of a distinction between "asshole" and "asshat" in this context, and I think the intent of your reply was pretty clear. Nor do I begrudge the comment--it comes with the territory of choosing such a username. But if you believe my characterization of that exchange to be false, my apologies. The link to the original reply was posted almost immediately after my comment, and so the context is there for all to see. I summarized it in a way that I thought would be entertaining to the audience.
So everyone can see that you are neither accurate nor entertaining -- thanks for confirming!
Link?
If you know a good way of searching through my entire comment history I'm happy to search for it, but it was some years ago, and I post several comments a day, so it may be a tall order to manually page through my history.
You can use the sort of official HN search engine like this: https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...
Awesome.
I loved that game.
I was a huge fan of co-op games and shooters, but all my friends wanted to play Counter Strike all the time.
With Serious Sam I could at least sometimes motivate them to play something I liked.
Thanks for your service! :D
Serious Sam was one of my uncle’s favorite games (along with Duke Nukem 3D). He was a hugely important figure in my life, and playing through his favorites is a good way to keep connected to my memories of him. So, thanks to you and your coworkers! Excellent game, excellent multiplayer, and very good memories.
you've no doubt heard it many times, but the split screen game play was very appreciated
That game was legit. You should be proud that all your efforts resulted in something truly great that people loved to play. Thanks!
Serious Sam was always a killer LAN party game. Not necessarily because it was the sexiest title of the day, or what anyone had planned. Serious Sam won the LAN party because even when every other game died under some driver issue, thermal issue, update problem, whatever, you’d load Serious Sam and it would just work. This continued through the later series too, hell of someone’s machine was completely dead in the water it would also reliably split screen and handled input well enough for split peripherals. The systemic parts of the game were truly exceptional on the reliability axis.
Serious Sam looked amazing for the shitty hardware it would run (FAST) on.
On a similar note, Counter-Strike never looked good, but was popular for a long time because it ran great on toaster PCs
Counter-Strike was/is popular because gameplay is awesome. Players stayed long enough that even "toaster PCs" could run the engine. It was never "cutting edge" even after major upgrades.
It was popular because it was fun.
I think this also explains the continued success of World of Warcraft.
There's a lot to be said for "good enough".
Also, the many speaker sound of “aaaaaaaaaaaaah” was delightful
I remember getting through some battle and coming on a room FULL of ammo of every kind. First time, it was awesome. But gradually ammo caches started making me say "Yikes!"
noone's mentioned the gargantuan size of the bosses!
playing co-op, running with your mates across the plains, blasting rocket trails into the distant horizon across the gorgeous clear blue skies at the biggest boss you've ever seen, with the pumping music thumping the whole way through, while you're trying to fend off what feels like hundreds of mobs ... I don't think I've experienced anything of that scale and intensity since ... one of my greatest of all time
lol, that's right!
"Wait, that guys is taller than BUILDINGS!"
Also every time you find a secret with some nice loot you just know that once you turn around it's going to spawn some more "presents" on you.
"aaaaaaaaaaaaah, yourself" was the best response.
In the late 90s, I managed the EA Tech Support website. The support/QA* teams would have massive after-hours gaming sessions playing Serious Sam. It was the only FPS that ran consistently on our work PCs, and it was a ton of fun.
* At least at that time, EA QA and Tech Support had a lot of overlap; support guys would be come in-house beta testers in the Summer when they were trying to get games done for the holidays and do Tech Support in the Winter around Christmas when the calls went up.
We used deterministic game play to implement multiplayer on the GB Color port of Vigilante 8.
The GBC Link Cable would pass 1 byte in each direction at the same time. It's a pair of shift registers filling each other up across the cable.
The game was locked to the GBC's frame rate. There was a lot work to update the screen that had to happen in each (effectively) V-Blank and if it was missed the smooth scrolling stuttered.
At multiplayer startup we passed our seed. To run it looked like this:
On frame A it reads the controls, and packs them into a byte and puts that in the transfer buffer. The transfer occurs while it renders frame B. At the start of frame C it has the local controls encoded in the byte sent on frame A and it has the other side's controls in the byte received in frame B.
It applies the controls to the game state and renders frame C. Local and remote controls are applied with one frame delay.
There was no frame delay of the controls for local play so if you ever lost in multi-player feel free to blame lag and me specifically if you need to.
I just bought this cart a few weeks ago (love the old GBC rumble carts!) and I was impressed that it had link-cable multiplayer. It's a really good game!
Thank you for the technical note on the multiplayer implementation -- that's really cool!
COOL! I'm really happy to hear that.
Croteam is such a talented team of game developers. I really enjoyed the Talos Principle (1 and 2) and they were some of the early pioneers of a completely custom Vulkan game engine for the first game.
I found devastating that in The Talos Principle 2 they had dropped their own engine and used Unreal Engine instead.
Me too. I think the main reason is that the lead render developer left, and switched to google to work on Stadia.
Ouch.
So now that lead is looking for a job? :)
Probably has buckets of cash, so no? :)
Not sure, Croteam never did come across as a rich company and I don't recall their engine sold well. Stadia work would've been a wage, a decent wage I'm sure but at the end of the day just a wage.
it's like every big game (studio?) is either microsoft already, or just not yet
like every gameplay stands as an engine, or a genre, or a meta-genre? like GrandTheftAuto?
but i was i trying to tweet something about getting lost in stocks and tickers and symbols
He now works at Roblox.
Talos 2 was a great game. Except the awful TAA. Made some of the meta puzzles almost impossible for me, because I literally couldn't see certain things.
I had to google TAA:
> Q: What is TAA and why should I care?
A: TAA stands for Temporal Anti-Aliasing. It's used to fix/clean up aliasing in games. The different kinds of aliasing in games are edge aliasing, shader aliasing, texture aliasing, temporal aliasing and specular aliasing. Edge aliasing makes edges of objects look jagged (also called as the staircase effect). Temporal aliasing is the shimmer of objects which are either small in nature, or which are far in the distance such as power lines, railings, stairs, balconies, antennas etc... You should get the idea.
https://old.reddit.com/r/FuckTAA/comments/oi0v86/taa_on_vs_t...
I found this disappointing too. I know they don’t have the resources to build their own equivalent of Nanite and such, but… On top of the performance issues, there were a number of things that actually looked so much better in the 2014 game. For example, the forcefields and water ripples.
Damn. No official Linux version :-(
https://www.protondb.com/search?q=talos
Gets a gold rating! I've been playing even the VR versions on Linux (through steam).
Yeah it works flawlessly on Linux, even nvidia RTX works.
I just found out the Talos 2 DLC is out on Steam this Friday!
Is this the same idea as "1500 archers on a 28.8" for Age of Empires?
https://www.gamedeveloper.com/programming/1500-archers-on-a-...
Yes. Both are “deterministic lockstep” systems. Many many games have used such a system over the years. Although it’s probably less common these days for a variety of reasons.
Factorio is a good example of a modern game that works this way.
Is this due to the heterogeneity of client hardware?
I think another issue with lockstep is that for it to work, each client needs all the input data from all the other clients to be able to simulate the next step, otherwise the simulations become out of sync. So how does the simulation handle both your input, and another clients input that comes in 100ms later? If you just pause until you get their input, your game is effectively pausing and running at 10 steps/second. For comparison Starcraft 1 ran at 42ms a step, or ~24 steps per second. You'd be playing in slow motion.
Games like Starcraft 1 would handle it by adding an artificial delay to your inputs (and all your opponent's inputs). Putting it into a buffer for future steps instead. If anyone remembers the latency option for "Extra high latency", this would increase the delay on inputs so that simulations did not have to pause while waiting for inputs from other clients. And then if any network fluctuations causes your input buffer of other clients to empty, you have to pause again.
In fighting games this is usually called delay-based netcode, and people hate it. It makes your inputs feel unresponsive, and in games that require fast reactions (like fighting games) gives you a smaller window to react.
Some games that used a determinism based architecture, will actually provide immediate feed back faked on the client and then some how make that work when the real stuff happens later. See Starcraft providing instant audio response when moving units or for Factorio: https://factorio.com/blog/post/fff-83 https://factorio.com/blog/post/fff-412 (second half of post)
Fighting games still typically use a deterministic simulation too. The difference is the good netcode, called rollback, works by using save states and rewinding the simulation to the time of your opponent's 100ms late input, and then resimulating the 100ms from there. You can get away with little or no delay on your own inputs.
I doubt that'll scale to games like Starcraft and Factorio, because in my example, you would need to resimulate 100ms of simulations in ~16ms. In a fighting game with 2 characters and not much else to simulate, that's a lot easier.
This is all way above my understanding, but:
Recently I’ve been playing Arma Reforger with friends online on a dedicated server, and when driving vehicles the collision physics are off, especially when driving fast, making the whole scene “teleport” forward, with frames lost and the vehicle reappearing in unexpected positions.
I wonder if this is related to this type of code. The game is technically out of early access, but there’s lots to be added still on the roadmap.
Not primarily. There were weirder CPUs in the past. Today's CPUs have largely converged on floating point operations. Basically just sin/cos/tan differ now.
It's possible to write fully cross platform deterministic code today, but it's harder to do and uncommon in libraries.
Writing state synchronization and using client side prediction is generally easier. Determinism is only necessary if there are bandwidth constraints.
Also it's easier to cheat when your local client has all the data.
Not so much easier but enables a different class of cheats in games where players don't have perfect information, even with other schemes there is normally enough information leaked that similar cheats are possible. OTOH using a deterministic simulation prevents a whole class of other cheats.
What class of cheats are prevented in deterministic simulation but possible with a dedicated server running a state synchronization netcode model? I don't think any...
Well, unless you don't trust the server.
Lockstep determinism has a lot of issues.
First, building a PERFECTLY deterministic sim is brutally difficult. Even for a single piece of hardware. It’s so so hard and expensive. A single bug “crashes” the game with a “desync” error. Making it work cross-platform was never really an issue.
Second, you’re bound by the slowest client. No one can go faster than the weakest link.
Third, you can’t stop cheating. Every client has full and complete world state. This is fine for co-op but a major issue for competitive.
Really the ONLY benefit to lockstep sync is it minimizes bandwidth. In the days or 56k this was a strict necessity. But these days client-server is almost always a better choice.
Wasn't that even worse before? Games like Warcraft and Doom played across Intel x86, Motorola 680x0 and PowerPC
Did Warcraft 1 and Doom have cross platform multiplayer back then?
I don't know about WC1/Doom1 but WC2/Doom 2 on the Mac also supported IPX to play against PCs in addition to the native AppleTalk support they had. The only time I ever saw IPX on a Mac.
Fun time. Developers essentially have to dig very low into the stack and deal with multiple architectures. The way to build great programmers.
They didn't have to deal with agile/scrum though. Clearly not real programmers ;)
Indeed, imagine all the cpu / compiler / different platform, all must behave the same.
This number actually shows how weird it is to have a unit cap in Tempest Rising.
Either you have the resources, or you don't. Don't cap for reasons of capping.
Yet games with 10x the bandwidth struggle to support that many enemies. It only just occurred to me: an increase of technological resources actually has a reversing effect on the efficiency and creativity of computer science. As bandwidth, storage, memory and compute has gotten bigger, there is an inverse response in the software that makes it slower, more bloated, and less capable, per discrete unit of resource. Call it the Benjamin Button software design effect
Do you have a number for "that many enemies"? If article is giving one, I couldn't find it. There are plenty of modern games that is multiplayer and supports enemy numbers that I would consider "massive". Or massive number of players if that is that matters.
I played it last night. There would be maybe a couple dozen enemies on screen. Which, for a dial-up connection, is a massive amount to communicate about in real time. The fights are structured such that it feels like 5x that amount, though.
Better known colloquially as software bloat!
Yes, this is known as Wirth's law.
The kind of game where I believe I moved backwards more often than moving forward.
Haha they even made a game about it, and its called "I hate running backwards". It's on Steam. I don't know if its from the same makers but it is in the Serious Sam universe.
Well, you and Netrisca were together, while those thousands of enemies, they were alone.
And the gun that seemed to shoot a fraction of a second before you clicked the mouse button.
...and, IIRC, desparately grasping for generated ammo to pick up while making said retreat.
Similarly, Factorio's architecture transmits (almost; notable experience is rail planner) just input events, relying on a lock-step simulation core.
I would love to work on a lock-step thing like that one day. Seems such a satisfying (and testable) design constraint to work with.
Their friday facts are very worth reading, not just for people that play computer games, it is interesting for any developer. If I remember correctly they had issues with parts behaving slightly differently on different platforms, which then lead to sync issues in multiplayer.
I have no experience here, but I guess the difficulty of such a strategy comes from ordering "events" in a game loop.
More than just ordering! Most floating point logic in compilers cares about speed, precision, then reproducibility (optimization can/will cause debug vs release builds to give different results)... Then you have to control inputs to the sim, you didn't think of (memory allocator, network timing behavior, random number gen)
Ahh, I remember playing a PC Gamer demo of Serious Sam as a kid. Even back then it was considered a "throwback" game to the days of old-school DOOM and Quake.
And now literally two decades have passed and its considered a classic in its own right.
Starseige Tribes was massive and ridiculously fun on a 56k connection.
There is a whitepaper exploring similar netcode concepts that was written by the Tribes developers:
https://www.gamedevs.org/uploads/tribes-networking-model.pdf
Quite possibly my favourite game growing up (especially Tribes 2!)
In fact, I downloaded Tribes 2 a while back and was playing against bots just a few months ago.
The game is dated but it was still fun... in fact, I've often thought of trying to remake it in Unity or something!
Maybe one day.
I never liked tribes 2 being a tribes 1 addict. The jet sliding in 1 was a happy accident and the developers lost the original physics code that resulted in the sliding being a controllable skill. Tribes 2 had an utterly different movement sensation and a “press button to glide” mechanic that just killed the game for me.
Tribes was brilliant. Skiing down procedural terrain (and watching other people ski up it), big maps, lots of players, in 1999.
Tribes was an absolute riot. Teams keep trying to make new versions but none feel like they capture the pure magic of the original.
The original is out there and free now. Though the community is gone :(
Isn't this exactly how Doom worked, long before Serious Sam?
But Quake went a different direction, sending the client updates about every entity they could see. Which had some benefits (players don't have to be perfectly synchronized) but also drawbacks (bandwidth proportional to scene complexity).
This is covered at the very end of the article, in the "Comparison with Doom and Quake" section. And yes, that's exactly the comparison and distinction the author draws, and they explain why Serious Sam would have opted for Doom-style networking (the notoriously high scene complexity).
Quake is vastly superior because the server actually run the simulation, for Serious Sam the server is basically a dumb relay.
It's not so much "vastly superior", it is more fit for its intended design and context. And certainly more sophisticated.
The server still runs the simulation for serious Sam and would kick any client getting out of sync (e.g. because they're cheating).
That was a fun game, I loved the demo.
I think some of the extrapolation might have also inspired optimizations made in CPMA mod for Quake3, which to this day, I think, has the best netcode of all games. You can actually compete with 100 ping, in it, which is completely unplayable in OSP or vanilla Q3. I don’t think any other game even attempts to be so optimized for competitive online FPS.
I loaded up the original Serious Sam The First Encounter about a year ago, and that game never ceases to amaze me with how much they were able to squeeze out of relatively low-powered 2002 computers. I had it as a kid, and I had never seen so many enemies on the screen at once.
Even now, the game mostly holds up pretty well and it still kind of impresses me.
Yeah, Doom Eternal struggles with like a dozen demons in one room and fps are immediately halved. It's such a pity. It's actually a decent game but if you can't keep somewhat consistent fps in a shooter game where every second counts, you'll just grind down the nerves and patience of your user base.
I know many people love Doom Eternal and other shooters with similar issues. Maybe I'm just sensitive to these kinds of frustrations.
Grand Theft Auto IV for the PC, in its initial release, would slow down pretty horribly if there was more than like four cars on the road. Even with a reasonably powerful computer, it was pretty bad to play (though updates did increase performance considerably). I remember when I first played it thinking "you know, Serious Sam came out like seven years ago on crappier hardware and it could handle way more stuff while maintaining a fast frame rate"
It's obviously not apples to apples, the models in GTA IV are much more detailed, but it was still pretty disappointing to play initially.
It's still my favorite GTA game, but I ended up playing through the 360 version my first time around.
Impressive writeup. The deterministic aspect is not super popular anymore, I guess, but this entire post does make me wonder how the networking layer compares to a slightly more modern multiplayer game.
It's very common in RTS. StarCraft 1 and 2 also use rolling lockstep.
Both those games are contemporaries of the Serious Sam series though. StarCraft came out three years before the first Serious Sam! But yeah fighting games and RTS games both commonly use deterministic sims.
Do I get it wright - the reason to implement a custom protocol above udp instead of using tcp was the benefit to have a certain amount of packets that didn’t need to be acknowledged?
Packetloss with TCP creates unnecessary latency for games.
The OS TCP stack basically will show data in a sequential fashion to the userspace code, this greatly simplifies things like a HTTP stream but in a game it also means that even if 9 out of 10 packets have arrived, if the lost one is the first packet then we need to wait for another roundtrip or more before seeing all the packets because the first one still needs to be re-sent before anything is seen.
With UDP and custom handling you can just proceed to render and show everything as it shows up, asking for a re-send will at worst cause re-simulation in an engine like SS's but for most parts the player won't notice unless something significant was lost.
For those that played Quake 1 before Quake World the difference was staggering, a Q1 session would freeze on packet-losses when the client tried to catch up(hard when a significant portion of the bandwidth was already used on the modem) whilst QW mostly kept on trucking with some jerks (they probably made other improvements also).
re: other improvements, it was also common for mods to change behavior between netquake and quake world.
Specifically I remember team fortress's pyro secondary grenade in QW being just pulses of AOE damage instead of shooting more fire particles that needed tracked, and the soldier nail grenade .. doing something different to not have as many flying objects.
That makes a lot of sense. Thank you very much for your profound answer which helped me a lot expanding my knowledge.
> But why do we see smooth movement and animation?
>
> Interpolation. Serious Engine interpolates between the current and the previous game tick based on time passed between. Try opening the in-game console (~ key) and typing this to see how the game looks and feels without interpolation:
>
> /net_bLerping=0
>
> It's kind of like playing a modern console exclusive.
That list bit made me spit my tea out, I laughed so hard.
Great game. A staple of our LAN parties, what, 25-ish years ago now?
I only played Serious Sam for about 10 hours sometime around release, but to this day the long scream of the beheaded kamikaze will cause me immense physiological distress.
> without interpolation … It's kind of like playing a modern console exclusive.
Ouch. It’s sad to see console gaming no longer be the place to go for highly tuned software.
To be or not to be, this is a serious question.
Croatia is one of those countries i can't point out on the map but routinely churns out amazing innovation.
Anyone else miss seriouszone?
Serious Sam was such a fun game - this was a really interesting write up too. I wish this kind of dedication to low latency and efficiency was more widespread.
totally agree. it was revolutionary at the time for the number of enemies on the screen. even at the time the vibe was very much, “croteam knows what they’re doing”.
really neat to be reading about this so many years later.
It still is revolutionary. Serious Sam 4 still doesn't take itself seriously. Best money i've spent on a shooter since ... Serious Sam 3.
I do miss the sledgehammer.
Agreed, have fond memories of the game! Just pure FUN for the sake of it! Really cool to read about the tech behind it so many years later, this is very well presented
Every italian knows the game thanks to Zeb.
I'm Italian and I have no clue who Zeb even is.
Zeb89, buddy you werent on youtube 15 years ago?
Always been, never seen/heard of.
I guess there's plenty of popular content on the internet or elsewhere I could name you and you'd be like "never heard of it".
Even extremely popular one.
i guess it depends on how old you are
37