Why the Japanese PS3 Version of Sleeping Dogs Uses the PlayStation System Font

This story is about a bug that I diagnosed and fixed back in 2012. I may have some details wrong, but this is how I remember it, in my own words.

Sleeping Dogs is an open-world action game developed at United Front Games and published by Square-Enix on August 14, 2012 for the Xbox 360, PlayStation 3, and Windows platforms. It is an open-world game inspired by Hong Kong action movies (but not exclusively Hong Kong films, since Tony Jaa movies were also an inspiration) and I would say that several films starring Donnie Yen, Chow Yun Fat, Andy Lau, and Tony Leung were major influences. If you enjoy video games like Grand Theft Auto and Assassin’s Creed, then Sleeping Dogs is worth a try.

The Japanese release is titled “Sleeping Dogs: Hong Kong Secret Police (スリーピングドッグス 香港秘密警察)” and the PlayStation 3 (PS3) version of that is the specific SKU that my story is about. A “Definitive Edition” came out a couple of years later in 2014 and that is generally the version that people play today, but it has been significantly upgraded: the PlayStation 4 has 8 GB of memory which is a sixteen-fold increase over the PS3 and you’ll see why that matters in a minute.

Sleeping Dogs uses a custom game engine written in C++ with an embedded SkookumScript interpreter used by mission scripts. Because game consoles of the era have a fixed amount of RAM and can’t (or don’t) use disk for memory swapping, a common practice for such projects is to use a memory allocator where each allocation is assigned to a memory “pool” with a specific budget. For example, Sleeping Dogs had separate memory pools for graphics, physics, AI, audio, and user interface, among others. Profiling tools would show you how much memory was being used in each individual pool.

One of the advantages to organizing memory allocations in this way is that you ensure having enough memory for the game features that matter most. Every megabyte of memory that you can save from efficient use of resources in one domain (like audio assets or user interface textures) means that you can use the memory elsewhere (more accurate physics meshes, bigger AI decision trees, more animation keyframes, and so on.) By enforcing budgets, you help to prevent situations where teams are fighting over who gets enough memory for their features at runtime. Another advantage is that you can reduce memory fragmentation and improve cache performance by keeping similarly sized allocations closer to each other, a technique called a “fixed-sized block allocation,” but that’s a topic for another article.

What happens when an allocation puts a pool over its budget depends on what state the game is in. In Sleeping Dogs there is a catch-all default memory pool that can be dipped into such that any other pool can temporarily go over budget without crashing (development builds emit an error message to be reported as a bug.) The budgets aren’t strictly enforced on platforms like Windows where there’s much more memory available (and virtual memory via swap), but for console platforms a budget overrun could cause an out-of-memory exception and crash the game, so overages need to be taken seriously. Of course, in the final production build of the game that gets released on discs, if a memory pool goes over budget and doesn’t cause a crash, nobody notices. The “high water mark” for overall memory usage is sometimes in an obscure corner of the game that most players may never see.

From what I can recall our memory budget on the user interface (UI) team was about 8 MB. We used the Scaleform GFx engine to render UI, and that consumed roughly 3 MB of memory before any assets were loaded. Generally speaking, 5 MB is actually quite a lot of memory for the kind of data that the UI cares about (map coordinates, shop inventory, action prompts, subtitles) with the only real exception being texture data which is huge by comparison. Like I said above, the less memory you spend on UI, the more you have for other gameplay features that may have a bigger impact on the overall quality of the game. Sleeping Dogs aggressively swapped UI texture packs in and out of memory to be as lean as possible, and that created a lot of work for our team.

The Xbox 360 has 512 MB of “unified memory architecture” RAM, meaning that the same memory can be used for either graphics texture data (readable by the GPU) or as main memory for gameplay data (readable by the CPU.) It’s a little more complicated on the PlayStation 3 where the memory is divided into 256 MB of system memory (for CPU cores) and 256 MB of graphics memory (for the GPU.) The upshot of this is that memory budgets for PlayStation 3 builds of Sleeping Dogs tended to be tighter all around and the UI memory pool was affected by this limitation.

One day during the final debugging push, we got a bug report that the UI pool was going over budget on the Japanese build of the PlayStation 3 version of Sleeping Dogs, and that in some situations this would cause an out-of-memory crash. We were close to shipping but I did have some time to investigate. After profiling the memory usage on that build of the game, I found that the Scaleform GFx engine was consuming a lot more memory than normal — up to a couple of MB, which is huge in the context that we’re talking about.

The obvious difference between the Japanese build and other language builds was that the character set — the number of glyphs in the font we were using — was much larger. A typical Latin style font might only have a few hundred glyphs, even allowing for accented characters, whereas a Chinese, Japanese, or Korean font can have tens of thousands. The target languages for Sleeping Dogs were the “EFIGS” languages (English, French, Italian, German, and Spanish) and Japanese, which was typical of console games.

The memory allocations for Scaleform GFx were a black box in that I didn’t have much profiling information about what they consisted of. I was well aware that GFx allocates texture map buffers for the character sets that it rendered, but these should reside in graphics memory and what we were seeing was that we were running over budget on main memory. I reasoned that perhaps Scaleform’s engine was allocating some memory for each glyph’s geometry and/or metadata and that this added up quickly on the Japanese version. This was an easy hypothesis to test: I made a build of the Japanese PS3 version with a smaller font and could confirm that the GFx engine was allocating less memory with the smaller font.

Online research into the issue revealed that this was actually a somewhat common problem that multi-region console games of that generation were running into. A typical solution was to create a restricted character set version of the font that only contained the glyphs that were used in the game. I was able to quickly parse our localization text database to determine a minimal set of characters, edit the font to create a minified version, and configure the game to use that.

This approach was effective but there were a couple of problems. Our in-game Japanese text was close to final, but somebody could still decide to modify it (a line of dialogue or the wording of a menu option, for example) and if they used a new glyph that wasn’t included in the minimized font, that would result in players seeing an ugly “character not render-able” box. Also, while the minified font did reduce memory usage, it still left the Japanese build with higher overall memory usage than the other language versions, and an out-of-memory crash was still a risk.

Digging deeper into the problem, I found a suggestion that I believe was in the Scaleform GFx docs. There is a PlayStation-specific option to tell GFx to use the built-in system font (called New Rodin) instead of loading a custom font. If you are a Sony gamer you’ve likely seen New Rodin in the PS3 menus (the “XrossMediaBar” or XMB as Sony called it) and lots of games, including Final Fantasy XIII. The docs didn’t say whether using the system font would reduce the overall memory usage and I was concerned that it would be less effective than using a minified font, but it was worth a try.

Enabling the system font — only on PlayStation 3, only for the Japanese version — remediated the extra memory usage almost entirely, much to my relief, and this solution was acceptable to the project designers. You may notice that New Rodin is a relatively wide typeface and it was a concern that this change would introduce many new localization bugs where text no longer fit on the screen, but this didn’t happen in practice. I think it helped that Japanese text tends to use fewer glyphs per word than English, so the shorter word width makes up for the wider font. It’s the inverse of the situation faced by studios trying to localize Japanese games to English where there sometimes isn’t enough room to fit the wider English text.

I was so happy to have found a solution that when the game shipped I specifically asked the studio admin if I could have a sealed copy of the Japanese PlayStation 3 version of Sleeping Dogs to keep as a memento, and they obliged.