Video codecs are a nightmare for game developers, but there's no solution in sight

"You very quickly realize there isn't a one-size-fits-all solution."

Sam Barlow, CEO of Her Story and Immortality maker Half Mermaid, has struggled with video codecs firsthand for over a decade. This type of software, used to compress and decompress digital video—anything from playing a splash screen at the start of the game or a cutscene that isn't rendered in-engine—is a silent, ever-present problem for game developers.

Frustration over this longstanding problem boiled over a little during an interview in April, when Sunset Visitor founder Remy Siu bemoaned how much time he spends thinking and fiddling with video codecs. Decisions such as opting for a codec that is better suited for PC or picking one that works on all platforms at the cost of delivering a lower-quality image plagued the development of 1000xResist. 

Now, working on the studio's next project, Prove You're Human, these problems are even more intensified, as the game makes use of "full-motion video" (FMV), including real-life footage of actors.

Related:1000xResist creator: the game industry needs a universal video codec for FMV games

Siu's experience made us want to hear from other developers like Barlow about their codec woes. What we heard—tales of cutscenes not appearing on Steam Deck despite working fine on PC, to having to deal with the peculiarities of multi-platform releases—showed video codecs present a bigger problem than they should, and are needlessly hampering the art of game development.

When do video codecs become a problem?

For context, the necessity of video codecs comes from the need to compress visual media of all types to avoid large file sizes. These transfers, forever, aren't flawless, as the process usually creates "lossy" files that lose some data from the original.

How much a file is affected, and in which way, varies wildly depending on the codec, as well as the hardware someone is viewing the video on. As such, figuring out which video codec to use for a game is reliant on a myriad of factors. 

For some projects, deciding on just one codec might be enough. Others may end up requiring videos to be presented in multiple formats, which can mean having to individually organize and validate hundreds of files multiple times. The latter, as Barlow explains, is a situation Half Mermaid knows all too well.

"Very quickly we found [that] if we're trying to push the performance, we have to go and use the native players on mobile, we can't get the performance on consoles, so we're gonna use Bink there... so you suddenly get to the point where you have all these different video players, every one of your movies is having to be exported into several different files, which, depending on how fluid your content pipeline is, it's conceivably a nightmare."

Immortality, for example, uses 10 hours of video. In-game this meant three movies' worth of post-processes, having color and sound delivered, tracking the right file version, and then putting that through the codecs. This resulted in hundreds of videos across different file formats.

Thomas Altenburger, CEO of Flying Oak Games, the studio behind ScourgeBringer, says that all consoles support H.264 hardware-accelerated decoding. On paper, this should be a good thing. But it all comes down to implementation.

"Each console has its own sauce to play a video stream, and it's never just writing 'video.Play()' and calling it a day," Altenburger said when asked about his experience with codecs. "You have to write code to boot up the hardware decoding, write code to ask the decoder to send you frames, write a shader to convert YUV images to RGB that the graphics engine can display, and write code to render that frame." 

"It's not that much of a big deal when you're developing for one platform, but it's the kind of thing that adds up very quickly when you have to port a game to three-four consoles, three desktop operating systems, and two smartphone systems."

Integrating a video splash screen at the start of your game, perhaps with a publisher or middleware's logo, or an animated intro, is the typical baseline for indies. But even these examples can be particularly complicated.

"When you're on a tight schedule, and suddenly notice that in your publishing contract it says 'you must include this video file unaltered at the start of your game,' the days or weeks you're going to spend to roll a five-second video are gonna be painful," Altenburger says. "Thankfully most publishers are fine with switching to a fixed image, but sometimes you're stuck."

On one occurrence where it wasn't possible to opt for a fixed image, the team ended up cutting the video into a slideshow of a hundred individual PNGs that it manually played back. "That made things terribly huge and unoptimized and kind of doubled the size of the game for just those five seconds, but it worked and shipped."

Video game codecs are more challenging than those used in video editing

In the case of games like Blippo+, which features six hours of live-action video content, the game's own structure sets a tough challenge from the start. Since the game simulates the experience of surfing "off-cable" channels, those six hours are actually divided into hundreds upon hundreds of one-minute videos internally. As developer and designer Mark LaCroix explains, this added time to production significantly.

"I'm working on this thing as postproduction is being done on the video content, so suddenly a new change has come, and it's like, great, I got to make the VP9 version. I got to make the H.264 version. I make the H.265 version. I have to validate all three of them, make sure that they don't have any compression artifacts, and I have to optimize them so they're the right size."

LaCroix was a video editor for a decade. He assumed his background would make editing Blippo+'s videos an easier experience. "I'm like, 'great, I will bring all my experience to bear, and this will be no problem. Other developers might have trouble, but I know what I'm doing.' But it's just such a different beast."

It doesn't help that, as multiple developers point out, there's a lack of documentation about best video codec practices in game engines and distribution stores. Alex Kane, founder of Pawsmonaut Games, found "zero mention" of what codecs to use in Unreal Engine in the company's documentation for playing videos.

"I released my game Aethus in March and it has several cutscenes which are video files in the game—they were encoded as H.264 MP4 files, which works fine on Windows," Kane says. "But little did I know that on Proton (the compatibility layer for running Windows-native games on Linux used by Steam Deck), the H.264 codec isn't included due to licensing issues, which means the videos don't play and at worst even crash the game!"

Some of these issues have eased as technology has evolved. Barlow notes that current-gen consoles having a solid state drive have meant being able to render video at a much higher resolution without facing too much of a restraint from the console.

For platforms with less horsepower, however, such as the Nintendo Switch, this is far from the case, forcing developers down a rabbit hole to make sure their games work properly. 

How deep does that rabbit hole go? LaCroix recalls leaving the Switch running for 28 hours, and getting up twice in the middle of the night to switch the in-game channels a bunch of times to see if certain configurations would work for long periods of time without crashing.

Is there a solution to the video codec problem?

Can the codec problem be "solved?" Veteran programmer Paul Hughes has attempted to do so with a high-performance, low-memory video codec called Chroma, but enticing as the idea of coming up with a modern solution is to him, he emphasizes that video compression is hard and comes with a minefield of patents. 

"Even if you made it a collaborative open source effort, you run into hurdles like NDA'ed platform SDKs, which can't be shared with everyone (Godot had problems with this for ages as they couldn't just stick the backends for PS5/Switch/Xbox into a public repository)," he says.

Several of the developers that we spoke with for this story couldn't go into much detail due to confidentiality. That's left them in a position where a single video codec may do the job well enough if the compromises aren't a dealbreaker for your particular game.

But the moment you start juggling different platforms and configurations, or attempt to push the standard in any way, however, it gets tricky. LaCroix points out that switching away from established video production tools isn't easy, given how well they're supported.

If someone were to offer a better solution specifically for games, they'd have to market to a segment of developers who are used to high quality, high performance, and efficient file usage, as well as incredibly wide hardware support, and then ask them to pay money for something that is slower and not as good, but may have all the features they want.

Pushing the predicament around video codecs one step forward, however, LaCroix argues that the limitations aren't just a nuisance to deal with during already long and busy development cycles, but can also create a ceiling of what is possible to do with video in games. 

"I think we need to get a better understanding of what our limitations are because, and video is the perfect example," he says. "We don't have people in games who know a lot about video because video is not well represented in games as a pipeline. So that means people with that expertise don't join the industry, companies don't hire for those skills, and toolmakers don't create things that are any different from making a media app for mobile or something.

"It's a spiral. I can see that in video because of my experience, but I'm sure that's true in all sorts of other areas of games where things are just the way they are, not because anyone chose to make it that way, but just because of inertia."

Game Developer has contacted PlayStation, Xbox, Nintendo, and Valve to ask if there are plans to address developer concerns over video codecs, and will update this story when they respond.

Update 5/12: This story has been updated to correctly describe the compatibility layer Proton.