Halo 2 with no matchmaking and only server lists would have been absolutely awful. I mean, the game itself would have probably held up, but imagine the nightmare of trying to find something besides either Lockout BRs 24/7 or something stupid like Waterworks Invis Snipers.

- No text -

John Hopson, A Lead User Researcher for many Microsoft and Bungie games, wrote a pretty good article about the time he tried to tell Bungie that Halo 2's matchmaking system was going to fail, and why he was wrong. It's a great insight into how you can draw the wrong conclusions by looking at data incorrectly.

I loved this article (and at the time I also loved the Halo 3 article in Wired that was referenced). I work at a software company and have been pretty heavily involved with design and user testing. We're all biased, and that shows up from every participant in the process. The designer is biased and notices the feedback that justifies the design. Often, users are not reliable witnesses to their own preferences--they may say that they consistently use some aspect of the interface that you rarely observe them using, for instance. A year or so ago, we implemented a pretty radical change in our design, and got feedback from a few users very late that they didn't like it, but it was just a few, and that's another thing: sometimes the most articulate and strongest feedback gets the most credence even though it's not representative. We recognize now that our new design is worlds better than what we had.

As with most things in life, the map is not the territory. Data is subject to misinterpretation.

This was a really interesting article, but I wish it had gone one step further and discussed the fact that there was a profoundly negative reaction to the fact that there were no custom lobbies. Bungie's forums were overrun with complaints (as they remain today) and custom lobbies remained an oft requested feature for years.

The article makes it seem like once the game was released, everyone saw the light and loved matchmaking, but that was not the case. Faced with that negative reaction to the real, actual game, Bungie still stuck to their guns, which I think is the even more interesting part of this, as it informs their behavior today.