QA stands for …

Quality Assurance. Not quality testing, not bug tracking, not feedback. Quality assurance. Assurance of Quality.

When you look at a painting, read a book, drink a fine wine, you engage more-or-less directly with what the creators made. When you use a piece of software there is a middleman. When you play a game it is dominated by the software. And yet the software you experience is indirect. The authors create source code which goes on to become the product you come into contact with.

The art you experience is conveyed via the software: the software is the canvas, the paint, the brush, the lighting, the lense, the camera and the projector while the actual artists themselves become draft sketchers because the art you see in-game is ultimately drawn and animated by the software.

If software were cuisine, the bulk of it would be created by a chef who can prepare and cook individual ingredients, but the meal would be combined, seasoned and presented by the plate and the cuttlery.

Most software disciplines go through an amoebal stage where applications are born of a single programmer. Obviously gaming is no-longer in that phase, but it hasn’t completely left the egg yet. So it’s been my observation that the attitude still exists within the game programmers that the game is born of them. But it also seeps upwards.

Unfortunately, game development has grown and this vestigial mentality can crush a product. A modern game is largely a corporate product. The concept is prepared, a design outlined and people brought onboard to make the product. Then, like a cell, it divides into the corporate product and the game…

The game is developed by the programmers. The design, the art, the animation, the sound, the gameplay are all processes and artifacts and controlled by the software. This would give the programmers a sense of proprietariness over the product, but in a more modern game development house the programmer is no-longer the rockstar.

This affects their relationship with quality control. Programmers are sufficiently displaced from the corporate concept of the product that they become focused on their own scope, and as games become larger and more complex, this is a bad thing.

In ye olde days QA were the guys who tested for defects before shipping. That’s not QA. That’s defect testing. The QA divisions have begun to climb the ladders of reporting structures in recent years but only a few companies seem to have realized that gaming – as a software development discipline – needs to promote QA all the way. They shouldn’t be an appendage, the clue is in the title: Quality Assurance.

If someone is stumping up $50m for you to build a game, even if it is your own board, surely your very first concern and priority should be ensuring that you are meeting the quality of the downpayment?

As happens when a discpline makes this transition, many developers still think that you can’t worry about QA until you have something for them to test. Wrong: from the outset they need to be putting things in place to make sure that quality standards are in place without hindering everything. They need to be making sure that your product is self-documenting not just in terms of comments in the code but in terms of stock taking. Is your bug database tracking variables that will allow you to detect and cure recurring problems in the codebase? Is your issue tracking robust enough to detect some implementation or tool that keeps costing time, without innundating the dev team in useless information?

QA systems can be a pain in the butt. Been there, felt that. But so is pretty much anything you tack on to a project late in the day. If QA is not just there near the start but breaks ground along with the code, then the QA team won’t be playing continual catch up and they can work on being a symbiont rather than a parasite. In those early days, when there’s not a great deal for them to test, they can expend efforts of honing their analytical tools and providing relevant and helpful facilities for the coders.

The QA team is where the buck should stop, so it makes no sense that they are some department over there or some sub-division of development. It should start with QA and stop with QA. The project needs to be QA’s baby because their role is to Assure Quality and if they don’t have control they can’t do that.

I’ve seen a few game companies try this approach when a project has started going sour. Which is why those experiments usually fail. That’s like blaming the band when you provided the music.

“Testers on top” is likely to sit wrong with a lot of coders. You can’t be answering to a bunch of test monkeys that first saw the project at alpha.

That’s not QA, those guys aren’t assuring, they are Quality Chasers in a perpetual uphill battle to catch up with the project.

Testers like are little more than line inspectors checking for obvious blotches. Real QA is about the project. Which isn’t just the end result but the process and the tools as well.

Software Quality = Effort Taken / Quality of Tools

The lower the quality of your tools, the more effort taken to achieve the same quality of software.

If you ground your project on QA it will be pervasive. The difference between pervasive and intrusive is how early and how well you integrate it Solid, concrete Quality Assurance systems should cover more than just your end-user product but your internal systems and tools and processes. That coverage should present itself as a utility to the development team.

Ultimately the success of a QA team/system hinges on one thing: it’s ability to assure the quality of the service it delivers to the teams operating under it and the people it reports to. In programming terms, a simple feedback loop. If you throw the QA concept in at alpha, it’s going to have a very hard time adding that kind of self-evaluation into its metrics because your team and your project already has a body of work with no useful metrics or data.

I originally wrote this draft back in March but could never quite figure how to finish it. I have some friends who worked on an AAA title that launched recently and did really badly and after hearing the inside skinny (which I don’t have the OK to share) I decided to dig up this little post… Their root problem seems to have been with QA being subordinate to the coding team and effectively being little more than testers and bug reporters. Most of the player complaints about the product are in their ticket database with a closed status of “won’t fix” from a coder. The actual quality decisions were being made by the coders or designers above them.

Full disclosure: I can only profess to hands on experience of these kinds of problems in a gaming environment at Playnet; in terms of gaming most of it is second hand discussed in conferences and in conversations/emails with other devs. It’s not a unique experience to gaming, it’s merely gamings turn to pass thru the process of maturing into a quality-based discipline.


Netscape hired 100 coders before they hired their first real QA guy. Explains a lot.

Of course I am coding games by myself but can’t have any testers on the bloody iPhone until Apple figures out how to do that. Unless I pass my phone around that is.

Very true. I believe in the Joel on Software approach myself. That is, the only person with the authority to close a bug is the one who opened it. The coder can say won’t fix till they’re blue in the face, it’s still there.

And I’ve heard that game companies do 0 unit testing. While I’m not working in the field yet, that seems….fucking retarded.

The Joel approach: That’s where ensuring QA is as integral a part of the development as the coding is vital – that’s the first evolution and it’s brutish; bringing QA in at the beginning on the team avoids them entering the project as aliens. Having a tester and a coder or artist fighting over an interpretation is no good.

As to unit testing, I can’t say: they’ve certainly been embracing it as far as I can tell these last few years.

I see game development as simply one kind of software development, and given that QA is not really embraced in the industry as a whole… it is not that surprising.
I have a friend that works in QA in a big company and he still has to fight every time that same battles you described. I was in a presentation he did a couple of weeks ago and it sounded very much like your text.
But I’m not really optimistic it will spread soon… I mean.. it costs extra, it goes against many programmers sense of being THE owners… it has too many enemies :).
In many places, you are lucky if you are even allowed to simply test things semi-properly before a release… oh well.

Whether it costs extra or not is down to the scale of the project.

If you aren’t putting together a one-shot team to create a $9.95 box game, if you’re going to have over a dozen coders and spend 2-3 years in development …

If you make “QA” the umbrella that contains project management, design and testing, to whom the coders are a supplier, then it’s not “extra”, it’s integral. That’s really the point of this post, QA should be integral. It’s extra when you add it. It’s not extra when you base your process on it.

You won’t spend more – at worst you’ll spend different and at best you’ll spend less. And if you don’t bastardize it, then your resulting product should require less support both in terms of technical support resources and post-launch developer time spent in bug fixes.

No matter how good your internal documentation systems or coder’s knowledge of code, the shorter the time between a line of code being input and subsequently revised the lower the cost – whether it is a feature, design or bug-fix change.

Coders are against it because they’ve worked with crappy QA adjuncts to existing projects. When you do that, you wind up with this bunch of guys who keep make you jump through all kinds of new and laborious hoops to try and determine if what you are doing is good enough – it’s like having the company bring in time & motion people.

A QA based system means the QA team are “on board” with the programmers from the beginning. It’s not just the coders under them it’s the designers and any one bringing in features etc. So part of their job is determining what causes delays and issues — which includes defective specs, defective documentation or request procedures.

But yes, it is hard to get coders who’ve worked on projects with shoddy or no QA to understand how it is a two edged sword.

I don’t know anything about game coding, but I do know something about software as an element of medical devices, since I work as Regulatory Manager for a device maker.

The FDA requires that US medical device software be validated. Rigorously. The only way to do it right if your product has any complexity at all is to design and build it with validation as its first performance task. I’m guessing that would be sort of like building a game that has as its first performance requirement, not having any bugs.

Yeah, I did not mean it *really* costs extra… I mean, for some managers, anything that is not coding and does not make you go faster, it’s “extra”. Not sure about that side of the ocean, but here most software companies take the Mc’Donalds approach to burguers: Fast & cheap so the customer buys. Quality is secondary, even if it will come later to bite them in the arse.

And given the lack of qualified manpower, they hire pretty much anyone with pulse, so people trying to do things *right* are not that easily found.

Between doing things right, like adding QA to the team, or throwing hardware at the solution, I know what many managers this side of the fence choose. The sad thing is that many coders also do, as well.

That’s really my whole point; adding QA to a team is fail because it will usually then be in conflict with the team or it will bog down the team with ancillary processes. Base the project on it, though, and it becomes a service.

BioWare’s Damon Osgood gave an example at AGC this year that I like to mention: Looking through some of their coding metrics he found that they a freakishly high percentage of bugs related back to a single header file. He went and re-organized/cleaned up the header file and the kind of bugs it had been generating dropped to background levels.

I’m fairly sure that most of us could figure out something like that, but that ought to be falling out of your QA. Not in your face, it doesn’t have to be a human performing by-the-minute code reviews and analysis.

As well as hounding you about your bugs and rejecting your “oops” checkins, your QA system ought to have the capability to help your coding team identify the things that are hamstringing them like “man we’re wasting upto 2 hours per coder per week just on builds”.

There’s also an interesting divergence between what QA is in the industry, and what the actual words mean.

Most places that have QA teams seem to just have them doing bug stomps… but I would call all of the following facets of quality:

* work efficiency
* code efficiency
* comprehensive yet comprehendable documentation
* ease of use
* enjoyable end result

I worked at Compaq, QA’ing iPAQs when HP took over. The QA leads were the ones that went to Asia and reported on which ODM device to pick for our next project. This all went very well under Compaq, but the first product we had under HP did not. They ignored our QA lead’s advice and went with the cheapest (highest profit) ODM handheld device.

Normally these projects have 300-500 issues. This one had over 3,000. Eventually, the suits said “ship it” and our QA lead said “no confidence”. They shipped it (under the T-Mobile brand) and T-Mobile pulled it off the shelf after a few weeks. After that, most of our jobs went to India.

In India they don’t talk back, they just do.

That way it would fail, but the reason wouldn’t be told in advance.

Leave a Reply

Name and email address are required. Your email address will not be published.

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

You may use these HTML tags and attributes:

<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <pre> <q cite=""> <s> <strike> <strong> 

%d bloggers like this: