Graev and I were talking about reviews today, and the subject came up about how reviews are so variable. We looked at Mists of Pandaria‘s reviews and saw a lot of “more of the same” in the ‘negatives’ section, yet it still manages an 83 which is a good score. Compare that to other reviews where a game or movie (Taken 2) scores horribly because it was once again more of the same, and blamed for “sticking to a formula.”
Then on the total flip side, games can be scored poorly, despite being good games, because they aren’t the same as a predecessor. That’s the “if it ain’t broke don’t fix it” mentality. If a game adheres to the formula, in many cases it scores well.
So you have reviews saying it’s good that something changed, or good that something didn’t change, or bad that something changed, or bad that something didn’t change… I think I have that right. Sometimes following a formula can be a great thing, or the worst thing imaginable.
Resident Evil 6 is getting slaughtered in the reviews, but Capcom has shipped (note: Not sold) 4.5 million copies worldwide. People are going to buy the game because they love the franchise, and despite its flaws that won’t change the game being good in their mind.
In general, it’s getting harder and harder to trust reviews. The reviewer could be having a bad day, or not like action rpg’s.  Reviews all come down to personal preference. The best anyone can hope for is that you find someone who, over the course of a long period of time, appears to have similar tastes and mood swings. Even then, the variability is extremely high.
So what do you guys think? Do reviews matter to you when making a purchasing decision, or 9/10 have you already decided before a game even comes out whether or not you’re going to get it? Personally, when I don’t already know for certain that I will or will not buy, I don’t turn to reviews; I take a much more organic approach. I think watching a game be played or reading a journal of someone’s playtime is far more informative.  Anyway, I’m curious what you all think.
There are a few gaming blogs like you guys that I trust that haven’t been sold out. Too many gaming sites and gaming blogs I think are on the take. That and/or just flat out fanboi that are going to rally behind a product just because it is from a particular publishing house or developer.
The problem is that a lot of people weigh any game under a 80 on a 100 scale as a bad game as well. I own plenty of games that reviewed in the 60s that are among my favorites games of all time. I also have plenty of games that ranted over 90 (or 9 on a scale of 10) that I feel completely were a waste of money.
I have $5 indie games that I rate way, way above $60 AAA games. It is all a matter of prospective. Do I feel that there are actually major sites that are getting money under the table to review products highly? I am sure. Is every review on sites like IGN, and Gamespot paid off? Nope.
The next problem is sites like Metacritic. I review Metacritic but I don’t usually let it sway my purchases over all. So many reviews have 0 – 3 ratings while the rest are 8-10. Rarely does anyone rate products in the middle range. You have the fanboi rating high regardless and the haters rating low just to be asses.
Even forums are a pain. You have to read though and try to pick out the actually intelligent arguments. Out of every 30 postings there is usually one that has substance.
It is getting harder and harder to find fair reviews.
I find I end up looking a game reviews in much the same way I look at movie reviews. I have a few writers who have a) judgement I find consistent, and hopefully close to my own, and b) can clearly articulate why they have the opinion they hold. I read them regularly enough that I feel like I can generally translate from their opinion to what I’m likely to think about the game.
I don’t worry about finding ‘fair’ reviews, but that might be a factor of most of my go-to critics (RPS, TotalBiscuit, etc.) tend to be smaller, more new games journalism types, and I’m expecting them to be biased and opinionated, but biased in a known way that I can account for.
I’ve been reading reviews of albums, bands, movies, comics, books, games and many other things for over 40 years now. As far as I’m concerned reviewing is an artform in exactly the same way the artforms being reviewed are and I judge reviews and reviewers on that basis.
I’m consumer of reviews as art, entertainment and literature, not as a buying guide. If I enjoy a particular reviewer’s work enough to read many of his or her reviews, eventually I may come to feel I know his or her tastes well enough to use them as a yardstick against which to measure my own potential enjoyment of the product in question and thereby those reviews might form part of my buying decision but that’s a by-product. What I come to reviews for is entertainment.
The kind of reviews that attempt to filter out the personality of the reviewer and achieve some kind of neutrality I find completely tedious. I’d rarely get far enough through one for it have any chance of affecting my thoughts on anything other than the reviewer’s lack of writing skills.
It is quite a coincidence that you made this post as I was just about to suggest that you broach the topic of validity of reviews such as Metacritic’s user reviews for AAA games.
It seems to me that Metacritic should just drop the numerical evaluation portion of user reviews in favor of a binary “liked”/”didn’t like” qualitative choice, as so many people are clearly using the former to purposely skew the average value in their favored direction as opposed to giving actual quantitative reviews.
Take MoP for instance, although I have no intention of playing as I dislike the precedent it sets for the industry, I in no way believe that it deserves a user rating of 4.4, with 121 positives (mostly 9’s and 10’s), 11 mixed, and 183 negatives (mostly 0’s or 1’s).
It seems that some degree of quality control is necessary if they are to maintain any degree of validity. Perhaps people’s scores shouldn’t be displayed by their reviews, and some metric employed to determine if the value should be included in the average such as at least 10 reviews submitted and being within +/- 2 SD’s of the median user reviews or the mean of the critic reviews.
@Neuro: I do find the Metacritic user reviews of indie games to hold validity and I do take them into account at least after reading their critiques.
Look at GW2 reviews across the board. Look at just how many gave it a score of 100 — a PERFECT score! I wonder what planet they’re on. Good game, maybe even great in many ways, but so flawed in some areas.
Then look at MoP: a formulaic expansion doing what works for Blizzard, capable of selling millions more copies, but can’t break the 83 mark.
I like Bhagpuss’s take on this topic a lot, and have a similar perspective.
In practice, when it comes to making purchasing choices, I usually look at one or two blogs and forums that I know share my tastes (ideally in that specific genre). I give forum/comment threads extra weight if I can find good back-and-forth discussions regarding the good and bad of a title. I don’t really pay attention to Metacritic or mainstream game review sites; they just aren’t measuring things using metrics I care about.
All that said, I don’t buy many AAA games at launch, so I’m typically not paying more than $15 (and if I’m really throwing money around, $30) even for a new game. As such I’m not usually too worried about making a perfect choice. Often I’ll get a game just because it’s deemed important to the broader game design conversation – i.e. I bought into DayZ early to have first-hand knowledge of that whole design experiment – or to support a concept, genre, or studio that I see as important, exciting, or risky and deserving of support.
Honestly I’d prefer to see less emphasis on game reviews and more on game criticism. That is to say, people writing thoughtfully about what games have to say and how they say it, rather than their value for money. Thus why I enjoy Keen’s ramblings about MMORPGs and how they achieve what they do.
What I find more helpful is to look at the technical complaint forums for said products. I just purchased Dark Souls: PTD Edition last night after much debate (25% off promo at Greenmangaming helped) but before I did I must have read two dozen reviews and watched twice as many gameplay and review videos. I agree that TotalBuscuit and OutOfEight as well as RPS are good places to look as well. I fairly trust TB though I don’t always agree with him but he usually brings up very valid complaints even in games he loves. RockPaperShotgun though I am a bit more leery of these days as they are starting to venture into that land of where it is hard to tell if they may or may not be on the take for some reviews so to speak. Though I have enjoyed their FTL coverage which I have been enjoying a lot lately.
One more thought. To me I try to separate Game Media sites aka Game Journalist sites from Game Commentary aka Real Gamer blogs like KeenandGraev. Just like regular journalist these days game media sites tend to be a lot more beholden to the almighty advertisement dollar and other agendas while you have game commentators like Keen and Graev that are passionate gamers and want to progress the industry not for money reasons but for a love of the industry as a whole. I think it is important to know what sites are just journalist and which are true game commentators. My money is on a review or suggestion by someone like Keen and Graev any day of the week over one from some hack journalist from IGN, Gamespot or similar site.
I also think it is important to read between the lines in user reviews as I often feel that people who are not intrinsically fond of a given genre feel the need to voice their opinions (for example fans of RTS games that review turn-based games).
For these people, I’ll let Mr. Stanhope reply…
http://www.youtube.com/watch?v=8k_trWYJpoA
I pay attention to reviews and personally score a lot of the entertainment I “consume”. What I sincerely wish though is if we could accept that 5.5 is the average in a 1-10 rating and not 7-8.
I remember decades back the UK computer game magazines exemplified a trend whereby the scores started at 80% and only increased from there; a lot of people still seem to think that way even now.
Beyond the numbers I do a lot of reading of actual comments, video reviews and forum browsing and try to filter out the industry astroturfers and blind fanbois to get to the genuine gamers who are trying to be as objective as possible.
I like to compare it to boardgames (www.boardgamegeek.com) where a review rating of 8/10 means that it is a very good game. A 7 is even a sign of a decent games and averages of the really good games are often between 7.0-8.0.
Reviews will always be useful as long as you take the time to read them and forget about the final score.
Seems like everyone has got reviews on the brain at the same time. There’s an article on the PA Report about this as well. There’s a video game research company called EEDAR that collects all kinds of data about what works or doesn’t work in games (I’m wondering if they are in the MMO scene however!) and other things that influence sales, which includes reviews.
It’s an interesting read and when they say reviews they clarify that they do mean an average of all reviews across the board. They acknowledge that some reviewers can be bought off or biased. They talk about correlation and causation in reviews. They mention some different psychological aspects of reviews. Anchoring was one where a few good or bad reviews can trigger more good or bad reviews from anyone who has seen the review because that review is set in their mind.
http://penny-arcade.com/report/editorial-article/the-power-of-review-scores-why-critics-have-more-control-than-we-think1
@Julian: Having worked for a website that does reviews, I can tell you that the content from a big site can’t necessarily be trusted either.
@Keen: Agreed, but then you’re talking about something slightly different, bordering on journalistic integrity and not the quality of the review itself.
Maybe we should expand the proposition to something like “Yes, you can always trust reviews assuming you know which interests the reviewer is tied to and as long as you ignore the final score”. Which I admit it’s much easier said than done and I’d say 99% of readers out there won’t (and shouldn’t, really) go to the trouble of following the trail of advertising dollars and parent companies.
I think the best course of action is, as it’s always been, don’t just read -one- review. Read as many as you can, that way you can minimize author bias and form a better appraisal of the game just by having many different samples.
Not to make an appeal to popularity but if you read ten reviews, both from independent and corporate reviewers, and they are all pretty much praising or panning a game then for sure that game rightly deserves to be praised or panned.
And sure enough as soon as I hit “Submit Comment” I was thinking the smartypants in the audience will immediately go “Oh, gee, if only there was a website that would put together and average all those review scores…” 🙂
I think it depends on the game and the number of reviews giving the same perspective. For instance, Resident Evil 6 is a member of a fantastic series that has received a lot of poor reviews lately that are most definitely hurting sales. Sure you’ve got metacritic but even then you have to deal with “bandwagoners” so to speak that love a game solely because it’s cool to like it. Then there are people who buy a game in a series without looking at a single review because they love that series, or the people that go to a physical store and read the game overview on the back and end up picking up a game like terminator salvation 😉
I don’t trust reviews from any company. Especially those like PC Gamer, IGN, Gamespot, etc. Those are so biased it’s ridiculous. I’ve seen them rip a game apart in the written review, then give it a high score. They are bought and sold.
I look more at player reviews and what friends say. I trust word of mouth and good gaming blog sites more than any review. I also look at their history to see their opinion on previous games of a genre, as they may just have completely different taste from myself.
Overall though, game reviews are worth very little, and any company that bases company bonuses on game reviews is really, really pathetic.
I think the Taken 2 guys don’t give a shit about crappy reviews – their movie is already a financial success anyway =P
every review is subjective to the reviewer’s opinion, you just have to sift it for information and make the decision yourself.
for example , keen’s hatred toward SWTOR dont impact my enjoyment of SWTOR and my own opinion that SWTOR is the best casual PVE and PVP mmorpg for me. The fact that they lost subscriber and have to go F2P do not have any impact on my enjoyment.
Some people love Guild Wars 2 to the point of fanaticm and irrational devotion, but one look and tryout in GW2 give me enough information to decide this one is not for me, no matter how much good reviews GW2 got and how much praise the mmorpg bloggers put on it.. same with TSW too..
so reviews or opinions are just another information for you to decide if that game is for you or not..
one man’s trash is other man’s treasure..
I find a reviewer I like and get to know, then I know how our opinions differ and decide accordingly
I have to agree with the others that reviews can not be trusted and that if it can be trusted its still the opinion of one individual.
Myself.. I watch actual gameplay footage preferably not directed by the makers or a professional reviewer. If I like what I see I go to forums and see what people are saying. Many opinions in forums not just from one person.
Last I check metacritic just to read obvious cons reviewers mention just in case any seem like a dealbreaker to me.
In any of those steps above I could have been sold to buy the game.
For instance I got worms revolution and xcom enemy unknown preordered for next week.
off topic:
Guild wars 2 is removed from my pc for now. I could play the beta nonstop, but in the actual game I had less fun and got bored after a few days.
Im playing planetside 2 and… its great!! You where right Keen. I’ve not played planetside 1 and the concept feels fresh to me.
I can best describe it as a larger scaled battlefield game where things look like mass effect every now and then. (energy shields, the purple technology faction, armor soldiers wear)
If you want a planetside 2 key find them on twitter (planetside 2 not the eu one) and ask in creative ways for a beta key. After a few tries you might get a key.
I don’t have time to mess with anything less than an 85 metacritic these days. Reviews matter, just ask FunCom.
For me, seeing actual play (in person or on the net) is the most helpful as you can see right away if there are any deal breakers (graphics not your thing, style of game play / content doesn’t suit you, etc). This is assuming we are talking strictly about electronic gaming. Other forms of gaming make this a bit trickier — how do “I” know if indie storytelling game Microscope is any good? I mean, the premise sounds good. Of course the author is going to try to sell it to me. So, I am left with: read reviews, or make a blind purchase.
Now, I know that reviewers have bias; some have more than others, particularly if they feel financially motivated. A decent reviewer “should” (imo) tell you the way something is and “then” how they feel about that thing. If you can pick out what you need from the review, say, the generalized mechanics of the d20 system or the control scheme of Magicka, then you are learning something about the product. Those things are not disputable, they are static and can only be interpreted one way. How a person happens to feel about the thing (the controls in Magicka suck! d20 is too crunchy — too many minis!) is a related matter, but those two qualities are distinctly different and separable.
Maybe reviews should go like this:
Review on XYZ. Here is the static information relating to XYZ. ** Here is my opinion on XYZ, and how it relates to the static information. (Maybe insert score here, but I really don’t see the validity).