Guest Rant

Veteran video game journalist Ara Shirinian gets a taste of his own medicine in this meditaion on measuring creative product with a yardstick.

The Futility of Ratings

Having written about games in various capacities for over a third of my life, I recently got my first taste of what it was like to be on the opposite side of the fence from where I usually stand. In the first four months of 2003 I planned, designed and implemented a map design for play in Unreal Tournament 2003.

Now, I am no stranger to software development, but this little project of mine was quite different. First, because I convinced myself that my future career would depend on the quality of the resulting product. Second, because it was my only substantial and committed attempt I had ever made in my life at tackling actual game development. Third, because upon completion of the project, I would receive ratings and reviews from actual players whom I had never met before.

My plan was to complete a map with a few novel features within the space of about three to six months. The beauty of the tools that I was working with was that they were all completely accessible to end users. If you buy a new copy of Unreal (whatever is the latest version), you automatically get the complete suite of tools in order to build levels and modify the game to your liking. This activity is tremendously more difficult than say, creating a custom park in your favorite extreme sports game, but it's also infinitely easier than writing your own engine and starting from scratch.

Game development, regardless of scale, is a far more discombobulated process than most people imagine. If you are a mortal like I am, you have to resign yourself to continuously modifying your implementation when the result does not fit your intended design. Mind you, the result never, ever exactly fits your intended design. If you are to make any progress within a limited amount of time, your plans must be malleable. You have to keep throwing out things that don't work, and you have to keep bending the ones that sort-of-work in order for gameplay to remain sensible.

After four months and somewhere between 350-400 hours of work, I finally completed a map that I was satisfied with. The great thing about Unreal is that there is a huge community of fans online. There are also several web sites that are designed for the primary purpose of advertising, distributing, and evaluating user-created maps and modifications. So I submitted my map to all the sites that I could find, and then I braced myself for elite hax0r rapeage.

So far, I've gotten ratings that varied all the way from a 5/10 to an 8/10. But my emotional experience in reading these reviews and the subsequent ratings was surprisingly enlightening. I felt like I could understand why some developers would want to call up the punk-ass reviewer and chew them out about the abysmally low score that their game received. Plus, after being subjected to this huge range of commentary, I began to think a little more deeply about the purpose and function of ratings.

The point of a rating is to somehow compress, summarize and encapsulate everything about the game into one metric that represents its level of quality. Everybody loves ratings. There is something naturally pleasing about knowing that product A is better than product B, by exactly 137 units. You are lulled into a sense of comfort. You feel satisfied by the authority and finality of a cold, precise number.

You can rate anything without too much trouble, so long as your metric is well defined. The futility in rating something like a game comes from the fact that not only is the metric completely undefined, but everybody is using a different metric! The situation is even worse than that, because giving a game one rating is like averaging, in some arbitrary fashion, an arbitrarily-sized collection of arbitrary ratings.

The video game is such a hopelessly complex amalgam of media. You have 2D art, you have 3D art, you have animation, you have interface design, you have gameplay design, you have enemy behavior, you have a system of feedback and rewards, you have sound effects, you have music, you have sound and music coordinated in special ways with gameplay. Each subcategory of a game can be broken down into several more subcategories, and each one of those can be rated in some way or another. How are you really supposed to rate a game, anyway?

Just because I play a game, and it feels like an "8," is that meaningful to anyone except myself? The only point in my life when I really felt like ratings were useful was when I was in high school and I read EGM all the time. They had the same four people rate all the same games each month. Over time, it was possible to gain an understanding of what each person valued in a video game by comparing their reactions. Only after knowing that did the ratings mean anything. Unfortunately, EGM does not do this anymore, as they seem to have a random collection of reviewers tackle each game. None of the other American magazines have ever had any rating system that came close.

Gamers' preferences are too varied, and games are too varied these days. It seems the best way to rate games is either in the aggregate, a la old school EGM and Famitsu, or to resort to actual reading / writing. Gamers should be less trustworthy of numerical ratings, and reviewers should be more sensitive to the fact that just because you don't like a particular feature, it doesn't mean that Johnny Gamer won't or shouldn't like it either.

-Ara Shirinian