TwMillion Blog

14 min read Original article ↗

3 Years of DDNet's Tutorial - 3 Million runs at once

Thousands upon thousands of replays synced up and played back together at the same time is mesmerizing to watch. With each one created by a human, you observe emerging patterns of behavior and strategies. This is what made this Trackmania 42k project special to me, and I've long wanted to make it justice in the game I develop tooling around, DDNet. I encourage you to check it out if you haven't seen it yet, the Trackmania action starts around 1:15.

For context: DDNet is 2D platformer that revolves around cooperative gameplay to pass challenging maps. It is a fork of Teeworlds, the PvP game that made DDNet possible. As the Teeworlds and DDNet are open source projects, they are a great playground for hobby programmers such as me to do fun projects.

Every single tool, from parsers and physics to renderer (except for general tooling like ffmpeg and Blender) was written by either me or Zwelf. For those wondering: Generative AI wasn't used in any part of this video or its process. In this blog I'll go over most stuff that made this project possible and I won't get too technical, though I might do that in follow-up blogs.

  1. Gathering Replays
  2. Map Choice
  3. The Pipeline
  4. The Renderer: TwGpu
  5. Hardware
  6. Artistic Choices
  7. Camera Workflow
  8. Some Statistics
  9. Unused Footage
  10. Closing Thoughts

Gathering Replays

Without replays, no video. In the Trackmania 42k project, the creators created a custom map and asked everyone to send in replays to be integrated into the video. This is a fine approach, but it is very time intensive and relies on interest from the community.

On the official DDNet servers, all gameplay data since 2017 is recorded and stored for future review. Millions of finished and unfinished runs are stored in terrabytes of data in a quite lightweight format lovingly called Teehistorian (a playable character is called a Tee). I'm pretty certain DDNet has one of the biggest collections of reproducable gameplay data among all games.

With access to the Teehistorian data, I'd not have to worry for a second about how many replays I'll get, because I'll simply have it all. So I wrote a proposal to the DDNet admins with details of my plans, and they graciously gave me access to the subset of data I needed for the video. Hurray :D

Map Choice

As I wanted to sync up all players at the startline, it made sense to me to choose a single-player map. In teams, only one of the Tees has to cross the start line to start the timer. The other Tee could be anywhere, so we wouldn't get a moment where all Tees concentrate in a tiny space.

Tutorial is the most finished solo map by far. Pretty much everyone knew it and has a finished run on it, perfect :) The Tutorial was released in 2022, so all runs ever created on it are indeed stored in Teehistorian. In the end, I used all Teehistorian files up until 2025-02-27, the third birthday of the map. These files amount to 756GB of data and roughly 3 million runs, great!

Full Tutorial map, 1 pixel per block

At first I always integrated all new Teehistorian files into my project. However, this also repeatedly introduced some new edge-cases that needed fixing. Replay botting also got increasingly popular around that time. I eventually decided that I had to set a fixed end date (2025-02-27) and never looked back.

I'm glad that I correctly identified the player Oak as the rank 1: A lot of botted ranks had to be filtered out. I for sure didn't catch all, but probably all relevant ones.

The Pipeline

The next image shows the different steps of the pipeline: Pipeline steps

Tooling

As stated in the introductory paragraph, all DDNet-specific tooling here was written by either Zwelf or me. None of the original DDNet C++ code was run, all required modules were our reimplementations.

Debugging

Every pipeline step took careful testing and debugging to get it into a workable state. The real work started, however, when all individual steps were "95% done". The remaining bugs were mostly subtle ones which I only observed after rendering. Debugging went like this:
  1. Do a render
  2. Observe bugs
  3. Figure out how to programatically filter the bugged runs from the demo
  4. Investigate the run files, perhaps the original server Teehistorian files
  5. Fix the respective part of the pipeline, pray to not introduce more bugs
  6. Repeat

Early image, weird hooks at spawn

Everything perfect, except for ONE hook

Camera mixup

No Tees, only projectiles visible

That is how I burned out a couple of times on this project. Whenever I thought I finally fixed all bugged runs and do a "final" render, something new showed up and required fixing. I did have the ambition to fix most bugs, even if they only affect less than a hundred runs. But each iteration took days to test, as I'd have to rerun the pipeline from an early point. I kept my head low and practiced my patience, fixing everything eventually.

For this reason I also kept this project secret for the longest time: I didn't want outside pressure to finish this project. That was for the best I believe.

My Renderer TwGpu

TwGpu is my own implementation of a TeeWorlds/DDNet renderer. I started it after learning the concepts of graphics programming in university and it does deserve its own blog post.

The graphics library I use is wgpu, a generic graphics backend over multiple possible adapters: Vulkan, DirectX, Metal, but also WebGL and WebGPU. WebGL and WebGPU are nice because this means it can natively run in the web once compiled to webassembly (here is it running in the web) .

Handling millions of Sprites

With millions of Tees running around, explosions and air jumps everywhere, many sprites need to be rendered. It peaks at 46.5 million sprites, each consisting of 4 vertices, one for each corner. Each vertex in turn has two floats for the position, two floats for the uv coordinates and 4 byte for the color. So 20 bytes per vertex and 80 bytes per sprite, resulting in ~3.5 GB of particles at the peak. For most of the project, the strongest utilized GPU had 3GB VRAM, hmm.

I needed to make the process, which generates the sprites based on the game state, streamable. To achieve that, the sprite generation now takes a callback that does anything it wants with the new sprite and returns a boolean that signals if the sprite generation should pause. The sprite generation can't pause everywhere in the process, but will do so as soon as it can. Now you can always tell the renderer how many sprites at most it should keep in memory. Also, this allows us to spin up many more threads to render, by limiting the amount of RAM and VRAM that one thread will use.

Multicam Rendering

A nice feature of my renderer is that the generated sprites are in game coordinates. Only in the vertex shader are they transformed into the camera space. This allows us to reuse all generated sprites for all cameras.

But sprites can only reused at the same exact timestamp. Overall, as I'm producing a video it would be great if I could precisely define which frames each camera should film. This way I can align all the frame timing in the camera and have fully reproducable frame outputs. To achieve this, the renderer now optionally takes a json file that describes all cameras. For each camera it specifies every single frame that should be rendered with it, it's timestamp as well as its position and ingame width.

With these well defined frames that should be rendered, they can also be easily split up along the time axis for parallelization :)

True Parallelization

Fully parallelizing a renderer isn't very easy, as it turns out. There was a time when a single batch of frames (~15 all rendered together using the same sprites) took ~2 minutes to render. Neither the CPU nor GPU was properly busy during the rendering step, at least not when going above 2 or 3 threads.

So, being confused, I went into the WGPU Users matrix room and asked what could cause this. The dev kpreid kindly took his time to read my rendering code and point out suspicous parts of it. In particular, kpreid noticed that I was creating a surprising amount of command encoders.

Every time a buffer needs to be overwritten, this requires a new command encoder. For simplicity, I created a new command encoder for the map rendering, for the four different particle groups and the game sprites. Each category needs more command encoders if they contain more sprites than the cap.

As it turns out, scheduling all those command encoders in the queue is significant overhead. Imagine 10 threads putting work in a queue, each thread's next workload having to wait for its previous work to finish. I'm not 100% sure how the queue works, it probably reorders them to an extent. Either the threads need to wait for some of the other thread's workloads, or only their own, but it slows everything down. With another rebase of that code, I minimized the amount of command encoders and now even with 15 threads the CPU is the bottleneck. Finally, a set of frames at the same timestamp only took 15 seconds at max to render. 0.07 fps isn't much, but in the end the rendering step only took 6 hours in total. 🎉

Hardware

At first I did everything locally on my laptop to get things rolling. Once I started scaling up the amount of recordings, this wasn't feasible anymore. My good friend Zwelf (also a DDNet player) gave me access to his server.

Zwelf's server:

Some time before I picked up the project for a final time to complete it, Zwelf got a new server:

Needless to say, the new server did make it even more pleasant to work with. The first server's best graphics card only had 3GB VRAM, which was difficult to work with at times. GPU comparison , CPU comparison

Camera Workflow

How to create the camera views and smooth moves was not a trivial problem to solve. There was no existing system to create camera views in DDNet, so I had to improvise. I turned to Blender, a 3D animation software that is both versatile and scriptable.

What I did was simple: I generated a huge image of the entirety of the Tutorial map and aligned it to the grid. The x and y coordinates thereby exactly matched the ingame coordinates. To aid in animating the cameras along the front of the wave of Tees, I wrote a script to import runs into the animation system. Blender's animation system with keyframes is very mature and with prior experience it was relatively simple to set up. All interpolation modes were available to me, as I could simply sample the animations at any time point with Python scripts. So the export script simply iterated through all cameras, wrote down their positions at every frame that was supposed to be rendered and serialized it to json.

Software Blender with camera views outlined on the Tutorial map

There were probably hundreds of different ways to go about and solve this. I'd recommend my method to a friend, tho.

Artistic Choices

In this section I talk about the artistic decisions I had to make in the order in which they came up.

Order of Tees gud conspiracy name?

The simple question is: In which order should the Tees be rendered? Should they be sorted by time/date when the finish was done, with the oldest finish on top? I quickly figured out that run duration is a better metric, with the fastest player on top. This greatly improved the effect of Tees washing over the map. Before it was kind of a mess.

Video Editing

I did all video editing in Blender. It worked well :)

Tee Skins

In DDNet, any player can select any skin from a big database with more than 2000 skins. They can also use skins from some community databases that are even bigger. However, many of those skins don't fit together well and many of the players playing Tutorial won't even have selected a skin yet. I agree that it could've been cool to use the selected skins, but that would've been again much more work. Instead, I simply took the standard vanilla skins and the skin is selected by player id modulo count of vanilla skin. Though the rank 1 does have the correct skin :)

Music

DDNet isn't a huge community, but definitely big enough to have active members I know who produced music as a hobby. In my previous videos I already contacted the community member (now also admin) louis and he kindly gave me permission to use his music. For this project, I wanted some different and also longer music. This time, ReD kindly gave me permission to use his music. Check out ReD's Soundcloud and Youtube channel.

Map Color Theme

One rather drastic change I did towards the end of the project is to change the base color of the Tutorial map. The change isn't exactly subtle, but I'm not sure many people noticed. The base color of the map was rather bleak and didn't match well with the stream of colorful Tees. I contacted the creators of the map, who explained that it does match well with the default skin, which is not among the skins I used for the video.

The original Tutorial colors

Modified Tutorial look

Some Statistics

My demo file consists of 18,000 ticks. DDNet runs at 50 ticks per second, so 3000 ticks are one minute and 18,000 ticks are 6 minutes. Exactly at tick 3000, so after one minute, all Tees cross the start line, meaning we have 5 minutes of gameplay. For the graphs I shifted the ticks so that tick 0 is where all Tees are crossing the start line synchronized.

Unused Footage

I left myself open which camera footage I would use, so what you see in the video is just a fraction of the ready-to-use footage. Before the last render I did strip the amount of cameras rendered down a bit as I did have a better idea of how it will look. Here is a Youtube link to the remaining camera footage, in order with timestamps. If you want any frame in higher quality or an entirely arbitrary frame anywhere at any tick, feel free to contact me.

Closing Thoughts

As much as this project exhausted me at times, it also always gave me energy when I looked back at what I already achieved. Programming everything was a joy partly because it was just a great challenge with interesting possible approaches. It advanced many of my and Zwelf's projects and matured them in satisfying ways.

The community's reaction was also heartwarming and I thank the few people who I showed it in its earlier stages and who encouraged me to finish it. After the video's release random players occasionally pinged me ingame and expressed their appreciation for the video, which is very kind of them. I want to encourage everyone to tackle small and big projects and see how they pan out, to also perhaps build bigger projects on top of them. Open source is a great space to be in with very kind people participating: Please don't forget that programming isn't neccessarily a hassle and can be very fun!

Thanks for reading. I hope you enjoyed this project information dump. Feel free to contact me for any questions at [email protected]