Why Video Games Have Launch Problems
playfab.comThis article completely ignores that single player, mostly-offline games have exactly the same launch problems.
The reason games have these launch problems is that publishers got away with it for too long. Having users who already paid for the game do the beta testing is much, much cheaper and more efficient than in-house testing.
I sincerely hope that steam's move to grant refunds (http://store.steampowered.com/steam_refunds) will sort this out. "Oops, 30% of our pre-alpha-published-as-final purchasers want their money back". Have one or two publishers file for bankruptcy over that and it might be a very valuable, albeit painful, lesson for the industry.
It doesn't ignore it, it's just a completely different issue. General awfulness of a game on a tight budget and deadline has little to do with backend scalability issues on large budget games...
The issue Augustine raises is why games have launch problems. He pins the blame on backend issues and ignores other issues. (I think primarily he was just using failed launches as a vehicle to talk about gaming backends, but nonetheless he argued that backends were the primary cause of failed launches.)
Another issue the article failed to address is that many games fail catastrophically when servers cannot be contacted even for online features that are not necessary for core gameplay; or for DRM purposes. This is a flaw in the front end that contributes to the same problem.
All those "causes" are primarily fueled by the following:
"It's a major threat to their business. So why does it keep happening?"
This here should have clued him into the larger context. It isn't a major threat to their business. Getting a refund on a videogame is becoming easier, but has historically been difficult. Try returning an opened copy of a game to any retailer. Even explaining that it is the game itself that is broken does not always result in an accepted return.
The consumer, rather than the business, is bearing the cost of a failed launch.
So Augustine is correct in a technical sense regarding how backend issues contribute to failed launches. However, it is the market as a whole that is the primary driver behind failed launches. There is little incentive to fix the failed launches regardless of the technical means by which they occur.
It pretty much does ignore it, pinning backend service issues as at least the primary if not the only major issue:
"So why does it keep happening? It comes down to the fact that most games are now online services, with updated content, special events, virtual economies, player interaction, etc"
As someone who plays both online and offline AAA games on console (PS4, Xbox One) and PC, I haven't seen any evidence that online-heavy games are more likely to ship with major problems than offline games.
For both types of games it is routine these days to get games on day one where you have the base game, you have the near obligatory multiple-gigabyte 0 day patch applied to deal with problems found between gold master and actual release, and then you still have obvious crash bugs, serious problems with animation systems, physics systems, etc. Connection issues with backend services are just one of a very long list of game breaking problems, many of which are seen in totally offline single-player games.
The whole "preorder" culture for gaming is a large factor in this problem, IMO. There's not that much incentive to ship a product that is solid when you've already gotten a large amount of the money you will get on release before the product even ships. This is even more evident in all the failed kickstarters where people have paid for alpha-level games that were never released or evolved much past where they were at the time of fund raising.
To a large degree the gamers are thus to blame for perpetuating this problem due to buying into the preorder culture (presumably in order to avoid loss aversion due to missing out on preorder bonuses).
Steam refunds was suspected to be the primary reason the Batman: Arkham Knight PC sales was suspended until the game was fixed.
> The reason games have these launch problems is that publishers got away with it for too long. Having users who already paid for the game do the beta testing is much, much cheaper and more efficient than in-house testing.
I'd say the main reason is that video games companies are usually run by a bunch of amateurs who are good at a few particular things (coding, design, creativity, and maybe even "hyping" things) but have shortcomings when it comes to engineering and delivering consistent, solid quality. Many companies dont have regular processes, and rely on employees forsaking weekends in order to deliver things on time.
I know it's usually not popular here to talk about the necessity of Project Managers, but the Video Games Industry is a pretty good example of an situation where more (i.e. better) management could help make things better compared to the current mess.
I do not think that games need more project management, but more people who share the lessons we've also learned in business and system programming.
For instance, we've had articles here about how beautiful the Quake 2 and Doom 3 codebases were. It's not that John Carmack isn't an amazing programmer (I am sure he is far better than me), but the techniques for readability and maintainability that he used in those games are just what anyone writing in a large codebase should be doing. The style he used is exactly what we demand when interviewing senior devs. But then game programmers look at it as if he had discovered cold fusion. But when all you see is games, that are written expecting that once the game is shipped, nobody is going to touch your code again, standards go down. Very different from what we do in business programming, where we know some systems might end up living 15+ years, so we have to write things planning for the system to keep evolving for decades.
If anything, online games and MMOs should be written better, precisely because they are far long lived, and will be edited more, than a throwaway that will just see a few patches if users complain.
You never saw a games company from the inside did you? Major game studios are just as much entrenched in their software development processes as any other company that size. Also, there is no single 'typical' game company. You have EA or Ubisoft which put 600 people on a single project, and on the extreme other end you have the one-man-army indie developer, and everything inbetween.
In the current game dev industry, there is no longer anything "in-between". The days of the medium sized games are long gone. It is either big winner takes all or very small indie under the radar.
I've seen two main reasons for this in my experience. The games industry is one where a 'chase the shiny' business mentality requires developers to frequently change their target platforms, development stacks, and desired core competencies for each project. Many of us are jacks of all trades, with little expertise in things like keeping a service standing under incredible load. I've seen developers go into this problem with a 'how bad can it be' mentality, and it never ever ends well.
The other side of things is where you actually have a team capable of delivering a solid, scaling backend service, who simply do not get the chance to do so because of poor production planning and overzealous development schedules handed down from on high. This is not an industry where the engineers get to define the development schedule or feature set. They can influence both, but on many teams the managers expect that crunch will solve all of their scheduling mistakes.
I think you guys at PlayFab are in a great space as this is work that is very difficult to pull off right and many, many developers will (smartly) choose to pay for a solution instead of rolling their own.
To address the point of "game developers don't know backends, and backend developers don't know how to build for games"... I want to think aloud about the question "how do you hire engineers who can work on the backend of a large video game?".
One issue parts of the games industry struggle with is simply coding competency. There's a tendency to hire newer grads and folks who will work for lower salaries, because people always want to make games. (And why not? Making games is awesome.) This leads to turnover, poorly thought out (or over-engineered) designs, lack of "common sense" things like load testing before launch (remember the industry standards you learned at college? if your college was anything like mine, you didn't), language soup, etc. And in my experience, the best backend systems are built by those with at least a bit of that experience already - rather than gameplay programmers trying to teach themselves what the CAP theorem is. But what senior engineer wants to work with legacy spaghetti code, or unlaunched promises (and the threat of future layoffs), when they could work at Facebook or Google or whoever?
At a previous large tech company I worked at, we built a games team internally to work on large scale platform stuff (pretty similar to what PlayFab is doing but aligned with said company's products). It was really cool - we got folks who were solid engineers but also ex-games industry, or avid gamers, themselves, advertising team openings via the videogames@ internal mailing list. We tried to combine the culture/fun of the games industry without the baggage and conditions.
But platform isn't content, and I recently joined a similar kind of team in an actual games company (which operates much more like a tech company than most, since our game is operated as a live service rather than a one-off downloadable release). Being able to work alongside artists, designers, narrative writers, sound engineers, event producers etc creates a really creative environment, and though I'm working on MySQL performance tuning and internal monitoring data pipelines, I get to hang out and talk about the new champion releases with folks at lunch. A nice balance, but for those who can't afford their own platform team, or don't have the carrots to lure us in, PlayFab seem like a neat alternative. :)
The difference with major online services like facebook and twitter is that games have a -very- big launch, and get a lot of users at the same time all at once. It's really hard to test for all that traffic in advance, as there will be so many configurations, (slow and fast) connections, hackers etc coming in all of a sudden. Its almost impossible to simulate the enormous variation.
Facebook, Twitter etc begin by a stable (code) base and then step by step scale up their systems. When something does not scale they can roll back or quickly fix it. They don't have to ramp up from a few hundred players to hundreds of thousands in a few hours like in the gaming industry.
" It's really hard to test for all that traffic in advance"
There's this thing, they're called dedicated servers. The reality is that game companies CEO's/execs don't want to pay for quality when they can push it out now get the money and patch later, aka. They put in the least effort. The whole thing is intentional at big companies, a combination of intentionality and incompetence at small ones.
I don't know what dedicated servers has to do with it?
I do not disagree on your point that it might be a combination of intention and incompetence, although I think most of them really do not want to see downtime and negative press at release date, when they sell the most. So next to intention and incompetence there is also a strong incentive for them to prevent this from happening.
But the orignal point I was making is that scaling up all of a sudden for an enormous amount of people/traffic is intrinsically very hard, so its easy for them to underestimate the effort needed, and pay deeply for that on launch date (and we too as gamers).
>I don't know what dedicated servers has to do with it?
First see the science, I can tell you the facts and figures and you won't reason to the conclusion correctly.
https://www.youtube.com/watch?v=PYmi0DLzBdQ
If you wanted to avoid "back end problems" (aka problems with 'stressing servers' because of your desire for locked down games where the server code is never released to the public) then you should allow gamers to run their own servers using server software like what games did in the 90's - problem solved. The corporate world basically stole gamers ability to own and run servers and since most gamers are 100's of miles away from these game companies they get away with it.
The free market simply cannot work when buyer and seller are 100's of miles away from each other, the seller has a huge advantage, all the outraged customers can't patrol outside the company and give the people there an earful for being corrupt criminal bastards.
Free market is a fantasy held onto by people who don't know much about how the real world works and gullibly believe the false ideological propaganda in the universities.
Protectionism for the rich and big business by state intervention, radical market interference.
https://www.youtube.com/watch?v=WHj2GaPuEhY#t=349
Crisis of democracy
https://www.youtube.com/watch?v=ZYFxtNgOeiI
Manufacturing consent:
I think the article left one important reason out: you always only hear about game launches that go bad, never about the ones that go smoothly.
Also: a game launch is like an explosion going off, lots of pressure on the first day. Marketing usually wants to concentrate everything on that one single launch event. Demand on launch day and the 2..3 days after can easily be 100x to 1000x higher than in the second week. You now have the decision to somehow spin up many times the backend infrastructure that you normally need, with the associated cost (theoretically achievable with a cloud service, but see below), even if it is technically possible to smoothly scale up against millions of players wanting to login simultaneously somebody needs to take the cost hit, this is usually when the finger-pointing between management and tech begins. Sometimes it's easier to 'wait it out', since the storm will be over after a few hours.
As for cloud services: Most game-servers have soft-realtime requirements which many cloud providers can't provide (at least with the cheap cost plans). For a web server running at a cloud provider it isn't much of a problem if you have 250ms load spikes or delays from time to time, for an action game it is catastrophic.
And finally: players that can't login will spend the time raging on the internet. If you can't get your iOS update on day 1: well, shrug, I'll try again tomorrow :)
> This way in the worst case, the player receives the item in their inventory and isn’t charged immediately. In practice, this has proven to occur very rarely and has been an acceptable tradeoff.
Except that now the player can spend the money twice in the race window...
That kind of sloppiness by game developers might be related to the launch problems they have.
Not having worked in game dev, it seems to me that some of the industry problems (high pressure launches, high turnover rate among developers, low ability to address or prevent technical debt) are responsible for at least some of the sloppiness. I'd think it really hard to make the right infrastructure decisions (and implementations) when you have a year or two to build a multi-million user infrastructure that has to adapt flexibly to any changing requirements from the game producers/directors/publishers and which will be thrown away a few years after that.
And having worked in gamedev, I can say you're correct. There's two kinds of studios that are affected by launch issues: The crazy big ones, who run into even crazier numbers that would make twitter/facebook engineers' heads spin... and the awful ones, that have the kind of issues you're describing and have launch issues because of management and/or technical incompetence. Though it's not mutually exclusive.
Also, it seems as if the incentive isn't there. Game studios know they can put a broken product on the market and still make money on it.
Which is why steam refunds make so much sense - as it's all digital, buying and refunding are the same couple of easy button clicks for the consumer. If this where to happen on the mobile app market it would be a game changer (because of the 100x more indie games there).
They caught up with the rest of the "internet-enabled" software. If it can be updated, the maximum it needs to be is acceptable, not (mostly) bug-free as in the old days. This gives lee-way to juggle quality with release deadlines.
As an outsider is it possible to do game development in an Agile manner?
It is. Some studios actually do this (e.g. Blizzard, High Moon, some had done it before they had been shut down). The problem is that Agile has been designed for a process at a service firm, one that does work for hire. As far as I understand, in that business you don't really want to ship. Instead of shipping you want to keep your customers engaged and paying your invoices as long as possible, preferably forever. This can also be useful for a firm that provides continuous service since there is no shipping involved too. And there is nothing wrong with it. But games have to ship. Continuous development without a hard shipping date is called "development hell" [1] in game industry.
From what I've seen Agile does not work very well here, at least on the engineering side. Design might be doing something similar to Agile on a small scale.
Of course it is. You can see the strategy of rapidly iterating on game mechanics first, for theoretically releasable, yet too crappy for the market, in interviews of old school devs: Sid Meier and Peter Molyneux built their first hits this way. Today we also see this approach in Early Access games on Steam.
The reason we don't see it everywhere though is because in AAA games, deadlines are seen as something so important that large parts of the game are built simultaneously. In your typical modern story-based action game, you have different teams working on different levels, and some people might spend all their time in just one or two levels: To get that level of parallelism, and have over 100 people working on a game, chances are that the development process will not have much to do with agile.
But many of the games that most people would consider great come from much love, refinement and iteration, along with relatively long development periods. This is why Blizzard always takes forever, and we have the concept of Valve Time.
Pfft... it's all just bullshit, pressure to release games too early from managers and hit quarterly profit targets.
That and the obsessive desire for control of everything.
Moreover is not these game have the necessity of being always online. Remember the bullshit about simcity having to offload computation to servers?
Scalability problem are not because the load of gaming, but as you say because they are obsessive about stealing every piece of info they can to better funnel users into premium.
Remember when people could host dedicated server? You would never hear quake having scalability issues or need to be saved from ec2
The reason IMO is game developers just don't care since it's a huge work to micro-optimize a game.
Last years when silicon reached it's limits, all kind of developers start to worry about the performance. The question "how to make a more advanced game?"'s answer is not git gud on hardware anymore. It's up to the software today (again).
For PC I get it, uncountable hardware variations. But for console games, it's inexcusable. You have one and only one variation of the console. Up your game.
This was talking about scaling backend services successfully and load testing them before launch, not about client side bugs.
Granted, I'm not sure why they included some games with client side problems in their list of examples.
I would say it has more to do with the fact that consoles now have hard drives and broadband internet connections. Games can be released as "good enough" to meet deadlines, with the bug fixes coming at a later date. This wasn't possible when consoles couldn't apply updates to their games.
You obviously didn't read the article. That's a completely different issue.
The article is mainly about server-side issues with the database, but those still reflect poorly on the client side.
Well if someone created an actually good programming language for game development we wouldn't have this problem...
Combination of C, C++ and a scripting language of choice are adequate and work well. It think business and design problems contribute more.