Have you seen this yet? Richard Garriott talks about what he felt his team did right and wrong in the marketing of Tabula Rasa. Richard points out that he feels the number of beta testers hurt the game. I’ve never really heard it put this way before but it makes sense. Beta testers are absolutely a consumer too and whether or not they liked the game or burned out on it has an enormous impact on not only whether or not they get it but if the people they know want to get it too.
Richard Garriott said, “And the people who did participate in the beta, we’ve had to go back to and say ‘look, look, we promise: we know it wasn’t fun two months ago, but we fixed all that. Really, come try it again.’ We’ve had to go out and develop free programs to invite those people back for free before they go buy it. So the beta process, which we used to think of as a QA process, is really a marketing process.†This goes back to what I’ve been saying. Beta testing isn’t about playing the game.
As a veteran tester I’ll be honest. Yes, I do absolutely look at what I’ve played in beta and make a decision if I want to get the game or not. I may have been testing it but if the game wasn’t any fun then I can’t afford to pay out of my pocket for that type of risk. The solution is simple. Don’t allow the testers to play the game; allow them to test it. Until your game is fun and functioning there really is no reason to open the doors and invite thousands of critics to play your game.
Looking over Mythic’s decision to close beta down for two months is really making a lot of sense now. When you look at it from this perspective it could be the greatest decision they ever made. They were letting a lot of people play the game and it was far from complete. Look at the decisions and changes they’ve made in the past 2 months alone and it’s apparent that maybe the game was either 1. Not much fun or 2. Not what it was being hyped up to be. Take down the beta, re-open with a smaller phase and do burst tests. Tackle one thing at a time then close the servers down. There’s no reason a tester needs to play beyond what currently needs testing.
Makes sense to me.
funny you just wrote about this. i was thinking the same thing about beta testing. i have done testing for games for awhile and it isn’t that fun. people always say “omg, you’re in the beta? you’re so lucky!”.
i want to say back, “yeah, it’s great to play a unpolished game. i bow down to the game gods everyday”. yeah, that’s not the case. it is usually a bad experience to beta test.
a rule i have is if i am really excited about a game i don’t sign up for a beta. i would rather play it when it goes gold so i know i am getting the best it could be (sometimes that fails to, but you know what i mean).
That’s a good rule. I tested potbs and it actually helped me want to get the game. However, as soon as I had tested the game a good bit I took a nice long break from playing it to avoid any burnout. I tested again for a while next phase then took a break.
I think you are right on here, Keen- I never understood how the devs could be sure the things that needed to be tested were getting adequately examined. I will say I think the last stage of beta, which is traditionally a stress test, should still be the traditional free-for-all. The game should be more or less gold at that point and the focus should be on keeping the servers running when the hordes start pouring in, and looking for bottlenecks.
Yeah, the open beta or final stress tests are more for marketing and server load testing than anything else. At that point the more the merrier.
“Don’t allow the testers to play the game; allow them to test it.”
Not possible, for two reasons:
First, betas aren’t just for finding bugs, compatibility problems, and other technical problems. Betas are also for testing gameplay concepts with a variety of playstyles, personalities, and community behaviors. That testers play the game too is vital to the game’s development. That said, it is possible to emphasize testing more in early beta than it is in late beta.
Second, testers will notice elements of gameplay even if they’re not directly involved. It’s like window shopping. Just because you don’t let customers in doesn’t mean they won’t form opinions about what they catch glimpses of through the window. The testers want to play, and they’ll use however little information they can find to make estimations of what the finished game will look like.
Developers can’t prevent testers from playing in betas, nor would they want to. The challenge is to moderate playing, not end it.
I think that much is a given Aaron. Of course they have to play the game. However, there is no reason to open the doors to say 50,000 people right away and throw them into the game to test anything they happen to come across while playing.
When that happens you run the risk of two things:
1. Having potential customers burn out of the game because they were allowed to faux test while “playing for free”.
2. Allow for far too many critics at a time when the game is incomplete. Which leads to poor marketing and a slow launch.
Here’s a very simple outline for how testing could work effectively.
– Internal testing with friends and family. Make sure the game works enough to bring the public in.
– Have a very, very small closed beta to begin with. Let them play the game some. Limit their levels. Perhaps let them play up to level 20 for a short period of time.
– Focus test specific content in bursts like WAR is going to be doing.
– Give those in closed beta a period of time to “play” the game unrestricted
– more burst testing.
– Open/stress tests, etc you get the idea.
Of course people have to play the game and all aspects of the game need to be tested. However, compared to CURRENT beta testing trends a lot less playing needs to go on and a lot more testing. Don’t let people play your game for 2 years while it’s incomplete (Like PotBS). Let them test it.
I wrote an article a couple days ago about why inviting a lot of testers matters… aside from just stress testing.
The basic point is that the behaviors of individuals and groups change as the size of the community grows. A community of 50,000 testers will act differently than a community of 10,000 testers. Different behaviors = different problems = different game fixes.
I’m not entirely disagreeing with you. There are a number of ways in which a smaller tester base is more effective, but there are ways in which a larger one is also effective. I don’t think there’s a right answer, really. Developers just have to pick their battles.
As someone else said, offering a free trial right at launch goes a long way in winning back disappointed testers (and there will always be plenty of those).
That’s very true as well and I hadn’t given much thought to that particular scenario. The numbers in the game do in fact change the behaviors of the community.
I can relate this most recently to PotBS. With what felt like maybe only 100 people online during closed beta the game was much different not only for testing but for playing than it is with the thousands online now.
It’s tough to pin things down. Each game will be different.