MMO development is not an art-form.

There is certainly scope for artistry in the programming of a game engine and its servers, but programming something as complex as an MMO is intrinsically science. Designing what the coders will write, that is largely art-form. But putting together the lines of code that become an inventory system or the loot system … very much science.

Writing computer programmes isn’t just sitting in-front of a screen and banging out the storyline. If you take the novella approach, you’ll end up having to jump through the same hoops writers and editors have to go through to get a book finished – ripping out whole chapters or the entire book to date, writer’s block, chapters that don’t fit your target audience.

Programming comes with a set of associated overheads which you must not avoid unless what you are really doing is playing roulette with your investor’s money. Beware the ides of “later”.

Before the first line of code is written, you should have at least decided what systems your game is going to have, and written specs for as much of each as possible and assigned priorities to them. And they shouldn’t be considered handed-off until the code/module has been delivered and a procedural method for verifying it has been provided — ideally in the form of a script or unit test, something that can be fully or partially automated so that you don’t have to keep going back and re-testing everything with every single patch.

The sooner you start testing code, the more solid it will be by release – more importantly your testing methodologies will mature with the project.

You think that’s going to slow down game development? Then you need to be fired, and probably shot. Anyone who says developing software properly gets in the way needs to be executed. Yes, developing software properly gets in the way of making two steps progress today. Then it saves you your three steps backwards tomorrow. Be wary of these “gets in the way” people – all they want is enough to make the investors believe things have begun rolling and will leave them alone. Sorry – but if you’re going into a project perceiving the investors as a monkey on your back, then you’re yanking someone‘s chain.

If you enter into a development project with good programming discipline and techniques, you’re going to have the ability to gauge your project, to begin planning ahead, estimating delivery time. You’re going to have materials you can use to liaise with investors and potentially get more funding or time from them. The week after they’ve forked out $10m for advertising and shelf space …. not a good time to be asking them for an extension, and to accuse them of pressuring you … deserves a flurry of kicks to the gonads.

Software is, by and large, quantifiable. Like ship-building, there are long stretches where the sum of quantifiables doesn’t add up to something you would want to test for seaworthiness. But if you employ good development discipline, you will be able to see and track progress through check-ins and unit-test development. The actual data you need to gather may not mean anything to you or the people you might show it to, it doesn’t have to. What’s important is “lines of code added, unit tests added, unit tests being passed”. Ideally, your coders will be receiving work orders from a ticket system and will be doing preliminary design as a set of sub-tickets which they task themselves with and turn in as they do their work.

Yes: it means paperwork. Yes: It means time spent doing something other than writing code. But writing good software is not about quantity, its about quality. If your programmers complete and sign off 100 well thought out and executed lines of code a day – 8 hours a day, 5 days a week – your product will be deliverable far, far sooner than letting your coders knock out 2000 “first pass” lines of code 14 hours a day 6 days a week to be tested by someone when its all finished or when there’s “enough” to do a big test.

Putting a long delay between when a coder writes a routine and when he has to debug it is a bad thing. If you want agile, fast development that produces good, reliable code, you need to document and audit upfront. And try not to have Coder A write an animation system in February and Coder C write the animation tool in January of the following year – it doesn’t matter an iota that you won’t be ready to provide animations until June of the second year, you should try and have closely-bound features/components developed by the same people within the same window of time. When a coder writes a system in January, he’s no-longer the same fully-fledged expert in it by June.

And you can never have too many stats about your development. You can drown yourself by trying to use too many of them, especially without understanding them. But you can never have too many – from how many lines of code were added or changed on any given day to how many unit tests exist and how often or when any of them failed.

You can’t generate this stuff later. And if you allow your codebase to be up in the air and intangible, then you are lacking something utterly crucial to the lifeblood of your business: assets. As a software development company, that’s what you should be treating code as.

The code is notthe magical property of the programmers that, with the wave of a wand, will eventually transform into your game. The code is the product that you provided your workers with tools and materials to churn out. It belongs to you and your investors and you should be as protective of it as a South African Diamond mine is of its shinnies.

Will your investors remotely understand you if you present them with information on how many new modules have been checked in or how many unit tests failed and had to be addressed? Maybe not, but you can educate them and you will have a mutual understanding between them and you as to what is happening to their money. More importantly, over the evolution of the project that same mutual understanding will serve as a backbone for two way conversations. The investors – and you – will be in a position to determine attainability of project goals, to make informed decisions about reducing or adding goals, and you will be able to communicate this in relevant and meaningful terms.

That only begins to cover what the right sets of metrics and management of code will do for you. Many of my readers will scratch their heads at this, because you’ve known it for so long you don’t understand why it needs saying. Knowing what’s failed over time lets you know what keeps failing or causing failures.

If the inventory system fails every time someone touches it, then you can surmise that it needs rewriting. Alternatively, you can just let your coders tell you “this is really broken, I’m going to have to rewrite it” and take their word for it. Who are you to argue with them? Well, you couldbe the man with hard cold evidence that projects written off the back of the current inventory system are usually delivered faster and with less non-coding time than other projects. Or you might have data that shows the inventory system generally breaks every 3 or 4 checkins and is clearly a complete nightmare.

Yes, that sounds suspiciously like planning.

Who knew you could do that with computer software?

There is, of course, some art and magic in writing an MMO. But you should focus on getting fundamentals developed first. Players will forgive you that flying mounts don’t quite work where they won’t forgive you that their inventories are unreliable or your network layer is hackable.

18 Comments

Good post. I used to be a development team lead (enterprise stuff) and this is something that I would have definitely forwarded to my team (and management) to read. I’m sure we weren’t doing half of what you are suggesting but we certainly would have wanted to move towards it.

It seems that many companies, especially in the gaming industry, are more interested in stunning visuals to sell the game, and just pray they can work out fundamental issues with the game after release.

The problem is that the time from release to mass exodus over issues is shrinking. Developers no longer have much time to work out fundamental issues before losing many customers, and getting a bad rep to boot.

I agree with what you are saying, as one must always have a good foundation to build upon. It is one of the things that was drilled into me during university. Make sure your programs foundation is fundamentally sound, and the rest of the program will be alot easier.

It’s the publi$her$ that are more interested in the stunning graphics, for one very important reason: publishers don’t play games – they sell them.

Publishers understanding “stunning visuals” so that is how they assess games. Devs that don’t produce them, don’t get the money.

Interestingly, I’ve received a few emails over the weekend relating this topic to the development of WWII Online.

Infact, I was talking about Vanguard.

Here I thought you were soliciting venture capital for a yet to be named MMO.

For me in the ERP world, I’m constantly asked to work some IT magic. I usually have to disappoint certain groups due to their lack of either documented processes or no standardized practices.

It is very hard to program by exception, and if you do program by exception you end up with something that works until the next exception.

I will not let the system be blamed for not working when in reality it is a departments, or business lack of controls that is the true cause of the issues.

I disagree. Programming would be an art if there were one main way to accomplish each goal. Even if we just consider the number of languages available, we can realize that there is no one “true” way to do everything. (Despite what the people fighting holy flamewars might say otherwise.)

This is, of course, the same reason why making these games (and making them stable) is so difficult. If I look at a problem and do X, but you choose to do Y, we may have completely different issues to tackle for our individual solutions. Going back to languages in a concrete example: If you choose C/C++, you have to worry about things like memory leaks, etc. If I chose Python, I wouldn’t have to worry about memory leaks, but performance would be a much more important issue. Part of the “art” is selecting the proper tools.

That said, this isn’t an excuse to avoid planning and just flail around until you get something that compiles. Take a look at a good fine artist and see how they work: they don’t just start slopping pigment on a surface, they often plan things out, make a smaller version, or even try other projects to get techniques down. Similarly, a programmer needs to understand their art and do the quivalent of sketch or cartoon to provide the proper planning.

In the end, I agree with your conclusion, but I disagree with your basic assumption.

My thoughts,

You seem to all assume that art = graphics. The whole “art” vs “science” thing is hard to wrap your head around.

Game development should be/ is, computer science, art, and SOCIAL SCIENCE. If you do a market study of a game concept (ie: will this sell?) that is social science. If you are plotting out a reward system, or teamplay tools and features – that again is social science.

Makes me wonder if there are any grad degree programs out there on game development from a social science perspective.

Trout

Well, that sounds truly nice, but that’s like fighting for world peace… if just all the other people understood and agreed… life would be so nice. Problem is, they don’t, and you still have to life in the same world where they live so…

Not that I don’t agree with you, but theory…reality… quite a gap in between. Not a excuse for not trying, but being blind does not help either.

S!

Brian:

I’m not precluding that there is some degree of artistry involved, but I think we’ve moved beyond the early days of pure garage development of online gaming.

Back-end systems for online games are increasingly complex software systems. I think, however, there is still a legacy of the garage approach that fails to see that much of what is being created is fundamental software development with very little to distinguish it from existing, well-understood software development disciplines.

Now, the particular blend of disciplines that are being combined could be seen as artistry – again I’ll credit that – but the underlying software development is increasingly just that … traditional, normal, software development. Game companies need to be wary of the mythos of bleeding-edge artistry where it doesn’t apply.

There is no reason why you should develop an inventory system in a vacuum because it is a “mode of recalling the contents of magical backpacks carried by a virtual avatar in a world of golden llamas and flying cobras who fight against rabid dingleberries”. It is a transactional inventory system, and any number of disciplines have excellent coding practices and standards for developing, testing and managing such a system.

Modern MMOs have many such components that ought to be perceived as stock elements and that ought to be developed as discrete deliverables to form the backbone of overall “artistic” development. That’s not to say that the game itself won’t have a fancy, artsy inventory front end with cool features. I’m talking about the underlying systems on which those aspects would be built.

What I’m speaking to is largely coding practice and discipline: many of today’s MMOs have all the clear signs of a team learning how to develop software – I’m not talking about learning how to write code, anyone can do that, but actually developing software is something different entirely.

First, I noticed a typo in my above posting; the first sentence should start out, “Programming would be science if…”

I’m not talking about content here, Oliver, I am talking about the same type of core programming you are. The problem is that there is no one “true way” to do everything from a programming point of view. You have to adapt to the specific situation and tailor the solution to fit it.

Take inventory systems, for example. The type of system and the requirements can change depending on your game. A turn-based game where you can only equip items between combats will have different programming requirements compared to a fast-action game where the players are expected to change equipment regularly and rapidly. The first game might allow for a slower system with more verification built in. The second game may have to sacrifice a bit of verification for speed. This is where the “art” side of programming comes into it: there’s no specific guidelines that say, “plug in some values and figure out which system you need to use”. What separates a good programmer from a mediocre one is if they can figure out what they can do within the limitations and requirements of the project. An off-the-shelf (or a well-document methodology) may not be appropriate for all situations. This is one of the reasons why I tend not to be overly enthusiastic about middleware solutions.

My further thoughts.

Hehe, I hope I’m not seeming adversarial – I think we’re roughly in agreement but that I’ve been unclear in either wording or nuance.

I would estimate the analysis you applied to the inventory system as “scientific”, if you will allow a crude black and white perspective for the moment :)

My proffered opine is that too many teams are allowing the default assumption or cry of “its art form” and that they resultingly allow significant portions of the project to spend an inordinate amount of time as “TBD”.

I’m opining based on taking my previous development skills to analysing various games I’ve experienced in the last year or so, coupled with my take on various articles and interviews and presentations on blogs/forums/GameDev/GamaSutra/AGC/GDC etc.

And I sense a little of the old rivalry that exists where one set of programmers said “Real coders were raised with an 8086” while the other says “If you never changed the screen color with a POKE you’re an accademic”; the “not invented here” syndrome when it comes to practices in the design and management aspects of software creation. Formal practices and methodologies still seem to be broadly shunned by many in the gaming sector as burdensome, and it takes the likes of Larry Melon to persuade people that they can have real value.

I’m not suggesting that – going back to our example – inventory systems be developed like banking applications, but there are lessons and practices from that discipline that could benefit the development of any game.

The syndrome I see is a lot of games suffering issues resulting from an overburdened conviction that the coders can pre-emptively account for a tiny fraction of what they have to developed due to the claim of art. I know one programmer for a fairly major gaming company who views Design Patterns as the spawn of satan; an affliction stemming from his belief that they are something used by web-game and Java programmers – who aren’t real game programmers. Most programmers hate the idea of unit-testing – they don’t believe they are being paid to test software, just to write it.

Networking and Internet applications suffered the same issue early on in the 90s, but the commercialisation of the Internet and immigration of programmers from other disciplines turned it around by the early 00s.

It also seems that many game-development teams are rapidly adopting their own concepts of “agile” and SCRUM development. Alas, twisted versions which basically say “get it built, then make it do what we need”. Vanguard seems, to me, the embodiment of this mentality, on the one hand shining with polish and elegant concepts but all delivered on a bed of torrid software.

Gaming has birthed its own discipline of practices and methodologies and standards for designing great games but it seems, broadly, to be lacking in good software development practices that would ensure that the games are built on great software.

A quick and dirty example: Code Metrics. Many development disciplines are subject to strict metric restrictions – many military and banking contracts will include several pages stipulating constraints for the code involved in the software they are paying for. It sounds a ridiculous concept for gaming unless you understand the reasoning behind it. It’s not aesthetic, its quite pragmatic. Meeting metrics specifications doesn’t necessarily force your hand in the final implementation but it can set a pre-emptive quality constraint on the execution of the implementation – and here I’m thinking primarily to issues like maintainability, testability, etc. Without metrics, coders can and will occasionally fall into the temptation to keep knocking out 7000 line functions in the name of efficiency.

There’s no disputing that in the world of gaming, performance is king, but its been my experience that programmers not invested with a suitable degree of accountability aren’t always aware of what is and isn’t efficient. There may be times when writing one huge 7000 ILOC function is an absolute necessity. But often times its just laziness – perhaps the coder didn’t want to pollute a C++ object’s namespace with lots of private member functions and wasn’t interested in writing lots of static inlines where he would have to keep saying object->{something} rather than just {something} in the top-level member function.

Until its finished, that gigantic function is a big “?” on the project schedule. Its also going to be a pig to test and debug compared to a more modular execution. Programmers who are accustomed to working with tight code metric requirements tend to better understand the benefits to themselves of things like design patterns and unit testing, and that yields code better suited to each methodology which in turn results in greater yields and leaves them more scope for being artistic without being Pollocks.

I suspect that many programmers in game development are like myself – they were presented with the choice of spending 3 years learning how things were done 3 years ago, or hitching a ride on the cutting edge. The UK educational system wasn’t teaching anything relevant to my interests when I left college (networking/communications/gaming) and I was presented with an opportunity to jump in at the deep end and both work towards my interests and learn hard-knock lessons in real-life, practical development.

As a result I’ve often been wary of concepts which seem to be the lifesblood of tenured MIT professors who’s concerns are publishing papers and selling books.

I’ll pardon your typo if you’ll forgive some of mine ;)

Yes, I think we agree in everything except terminology. Instead of the title you have, I would have said, “MMO development IS an art form… but that’s no excuse for poor planning.”

“Science” means that you have a set of observable, repeatable theories and assumptions. If MMO development were a science, there would be one correct, repeatable way to accomplish each task. Instead, we have to evaluate the conditions carefully, apply good judgement, and produce a system that fits the individual situation.

I think the real guilty party here is the people that say, “it’s an art, so I can ignore planning,” that you are referring to here. Just because something is an art doesn’t mean you don’t have to plan. Go ask a fine artist to paint your portrait. They should take the time to plan out the paining: do sketches, maybe plan out some color schemes, get verification. The portrait artist that just starts throwing random paints on the canvas in the name of “art” probably isn’t going to give you the portrait you want.

(This whole issue is confused because it is acceptable for an artist to start throwing paint on a surface, whereas this usually leads to crying in the world of programming.)

In the end, the important part here is that developers need to do proper planning. I think that’s something that most experienced developers can agree on. I also suspect you’re right in that the less formal backgrounds of many game programmers tends to lead to the “hack it until it works” mentality that can often cause so much trouble, especially in a project like an MMO where things like stability and maintainability are much more vital to the success of the project.

So, would this be a good example. They are lisencing the foundations to build upon. No reinventions of the wheel.

http://www.nerfbat.com/2008/03/05/38-studios-licenses-unreal-engine-3/

This looks like it will allow them to focus more on the content, look and feel, an less on the basic mechanics.

Breed, buying off the shelf doesn’t mean you’ve addressed the process issues I’m talking about :)

Brian: I actually think there’s a lot more science to the process than just planning – there are plenty of games that have had good, scientific planning but have then failed on the execution of the development process. The planning stage can be mired in artistry if you don’t have processes that tightly integrate planning into the development of as much of the fundamental software platform as possible.

You can reach the old IBM extreme where everything is over specified and engineered to eliminate the “risk” of programming art and wind up with something like Horizons.

If your planning introduces good software engineering practices into the programming phase, then the planning can afford to be more dynamic and the overall development of the game can be far more artistic because as your application evolves your team will have a perspective on its progress and feasibility.

If you limit your science to the planning stages, then you will have either blackbox or bursty development – either all of your software is intangible to all but the coders until a catastrophic deadline or you have waves of deadlines each of which involves large amounts of wide-ranging testing and evaluation that fails to completely proof the software being tested, making it an expensive mis-use of time (as some people have assumed is what happened with the aborted Warhammer beta).

Better modularity and focus on your software development should result in an ongoing testing process that starts with the programmers validating their own code to specifications. It lends itself to true agile development, because from the get go the code is an asset rather than being abstract.

Most programmers are averse to having to write code that works – they’ll explain it away a variety of ways but ultimately they take it as an affront that you could think their code might not work as expected. Developing to a process, however, is wonderfully liberating. When the production team come and tell you that isn’t what you want, you can pull out the specs and sign-off documentation, the unit tests, and say “It does exactly and precisely what it says on the box. Maybe if you tell me what you actually need it to do I can make it do that instead” – instead of “but that’s not what I thought you asked for” :)

I agree with Oliver, breed, just because a company license something doesn’t mean they are licensing the right thing. Does the tech fit with what they want? There are limitations on many graphics engines that become important. However, 3D graphics engines are one of those things that doesn’t make much sense to re-write every time. The engines are becoming commodities.

Anyway, back to my conversation with Oliver: your problem is that you’re speaking in absolutes. Some programming is definitely down to a science. For example, writing joystick drivers is down to a science at this point. There’s not a whole lot you can do to improve them, short of writing new drivers for a new console.

And, I’ll agree, a lot of programming tasks are more science than art. Writing a 3D graphics engine, if you weren’t going to license one, is mostly science. However, the people that push the state of the art are engaged in more of the “artistic” side in most cases. Real breakthroughs, especially during the time when 3D engines went from DOOM to Quake 2, came from people (mostly Carmack) thinking of things in a different way. A lot of the innovations came not solely from evaluating things from the scientific point of view, but rather from a spark of inspiration or cleverness. This is where the “art” comes from.

Now, some MMO programming is the same way. Network programming, for example, is a well-understood discipline at the core. Moving bits across the network is a science and there’s not a whole lot of room for artistic interpretation. As you get away from this base, you get a bit more room for the “art” of programming: at the next layer up, how do you optimize generic traffic? Mostly science at this point. How do you reduce network traffic for your game? A bit more art, but still mostly science. How can you treat a distributed object like a local one? Again, a bit more art comes in. How can you design the game to reduce network usage? This is perhaps where the most art comes in.

Once you get to things that are specific to the game, however, you have a lot more of the “art”, in my opinion. An inventory system is pretty well understood because multiple games have implemented it. But, what about squad-based movement of multiple units under a single player’s command? You can get some information from single-player games that have had this feature, but now you have to consider how to optimize that under client/sever restrictions. What about allowing the player to write A.I. scripts to control their characters even when they’re logged off? Few games have implemented anything like this, so you’re going to have a lot more of the “art” aspects come to the forefront. Some parts of this, such as implementing a scripting system, are almost down to a science, especially if you embed an existing scripting language. Many other details aren’t quite so set in stone, so they’re not quite 100% science yet.

The type of game is another thing to consider. WW2O, for example, had a lot of other, single-player games that could be examined. If someone were to create a computer-hacking simulation online game, there are fewer types of this game than military-inspired shooters. Implementing a lot of those systems is going to be more of an art than a science.

Anyway, there’s my arguments. Again, this is not to argue against planning or good engineering practices. But, I don’t think it’s fair to say that all game programming is 100% science and there’s no art to it at all. I think there’s significant art to it, and we have to understand that this brings a non-trivial amount of uncertainty to the process. This is why good planning and practicies are so important, IMNSHO, in order to reduce the impact of this uncertainty.

10 years ago, game development – especially programming – was an artform: to achieve in 20 cpu cycles what ought to take 100 or 250.

The case I’m trying to argue is that today’s gaming software development could benefit from re-rooting itself in good engineering – perhaps that’s a better term than science.

I’m freely admitting that gaming development is dependent on large amounts of artistry – especially in design. But its my belief that the development – and particularly the programming – of the software involved ought to be rooted in the fundamentals of good software engineering.

Not to stifle the artistic component, but to promote it.

Taking your squad-based concept for an MMO: you should be able to design a broad outline for the system, have the coders go away and produce specs to achieve the broad goals and implement fundamentals for the system such as a scripting engine for commands, state engines for executing them, etc.

Now, the coders don’t have to come back with a completed system. They can return with a system implemented to N% but already tangible. It can already begin interacting with other systems and those interactions can begin to be tested and secured as assets. That doesn’t mean they are immutable.

Now on top of a tangible system which may potentially allow experimentation, the design team already has some notion of the costs associated with this area of development and the timeframes available to them. If it took 6 months to get the basic commandable AI system integrated and there are 3 months until beta, the design team will know to keep this simple.

If it took 2 months and there are 7 months remaining, they can afford a high degree of creativity in trying to perfect the AI command system and get it as well developed as possible.

And if you have good engineering practices in place, the programmers actually get a bit more room for creativity. For a start, the more of your software that is quantified and secured as assets, the shallower crunch time should be; risk assessment should be much easier so that less paranoid “risk-aversion” rigamortis sets in and the coders retain flexibility.

Perhaps my title should be “good coding isn’t art-based, and game development needs more good coding” :)

I think the difference can be seen in our own expectations of the software. When buying a productivity application, we expect it to work as designed with very few bugs. When buying a game for the PC, we expect to be installing bug-fixes for the next year.

Obviously a well-structured design for the coding can be achieved. Just look at the titles that are developed for PC and a console simultaneously (KOTOR, Jade Empire). Those are released on the PC (and the myriad of hardware configurations that includes) with few, if any, evident bugs.

That is primarily due to the requirements for a console. No patches for the console. You only get one chance to release the game. Not fix it over x number of months.

You’ll find that if you are not forced to do something correctly then you won’t. I see this daily with the way businesses operate.

Leave a Reply

Name and email address are required. Your email address will not be published.

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

You may use these HTML tags and attributes:

<a href="" title="" rel=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <pre> <q cite=""> <s> <strike> <strong> 

%d bloggers like this: