caproven

The impossible task of rating media

cityscape screenshot from Kiki's Delivery Service

Last night I had the pleasure of watching the Studio Ghibli classic Kiki's Delivery Service with a couple friends for the first time. As it's a nearly 40-year-old kid-friendly film, we didn't feel the need to analyze every minutia of technical delivery and storytelling. A few gripes with abruptness of the ending but not enough to detract from the impeccably charming and immersive world Studio Ghibli is known for painting. Great film overall.

This got me thinking of how we choose to rate media, an impossible task rife with subjective views, elitism (critics vs. us plebeians), and no standard framework. Who chooses to rate in the first place, and which inherent skew might that introduce? I've talked at lengths about this topic with a former friend in college who wrote a full analysis and review on every single player video game he completed, although he had much stronger insight on the matter than I.

He advocated for a 4-point rating scale interpreted as:

Score Did I like it? Do I see how others may like it?
1 No No
2 No Yes
3 Yes No
4 Yes Yes

In fairness this individual would also issue a traditional 10-point scale rating based on a fine-tuned rubric, I believe a weighted system averaging the categories of story, characters, musical score, gameplay, and a small bucket for personal favoritism. The 4 point scale however attempts to remove our seeming bias towards mediocrity being gauged around a 7/10. To the reader, ask yourself: when was the last time you rated or saw a rating of 5/10 or lower?

I've come to the conclusion that most people are simply horrible at objective analysis. I myself am guilty of this and am easily swayed by the "vibes" of media I consume. Christopher Nolan's Interstellar is one of my favorite films I'd happily advertise to the world, but it is no perfect film. I just thought it was cool, cool enough to overlook technical flaws with ease :).

If people are incapable of objective analysis (or just don't care), you end up with these heavily skewed rating averages - TV and film rarely go below a 7/10 and product ratings such as on Amazon may average nicely but are largely composed of 1/5, 4/5, and 5/5 star reviews. The 4-point system is not more precise than other scales (in which case we should all use N/100 scores). It's primary benefit is the "Did I like it?" "Do I see how others may like it?" dimensions acting as an in-laid rubric demanding analysis through a lens other than the individual's own experience. Semantic mappings for otherwise ambiguous mid-range values greatly helps with conformity - no confusion of "what warrants 6/10?". I've found this promotes healthy banter about media instead of devolving into "I loved this and anyone who doesn't is stupid" defensiveness (which I'm totally guilty of).

It's worth addressing that ratings and recommendations are not the same, yet I've anecdotally seen them folded into the same concept. To this I would ask which rating threshold implies a recommendation? The 4-point scale does not entirely mitigate this and I do not think it is possible to do so. The "Do I see how others may like it?" question intentionally does not define "others"; it could be everyone, the intended audience (as you perceive), or a group you expect would appreciate it. I intend it to be as precise as you care for it to be and concede that as a heavy flaw. I don't have a better proposal which doesn't bloat and over-complicate the scale.

Media is intended to connect with its audience and stir emotion, which is further reason for rating skew. A product is often rated on quality and a direct measure of usefulness. A movie in contrast as a form of entertainment has more subjective dimensions for the moviegoer to scrutinize the experience over. I admit this may partially explain why lackluster media tends to average around a 7/10 - who is intentionally making shit media? Who would watch and therefore review a shit piece of media? Media has become increasingly expensive to produce and consume which leads both sides to be risk-averse. Producing a movie for >$100m requires much more risk than I the viewer take in potentially wasting 1-2 hours of my time, but you get the point.

Separate from ratings I would like to address the misleading nature of averages. Example-wise this is easier to frame around a product review - say you have two competing products

Both products average the same but the rating distribution tells its own story. Product A may have rampant manufacturing faults, and Product B may work but is cheaply built. The TV/film equivalent might be a divisive film (niche audience, taboo subject area, etc.) contrasted against a film everyone thought was passable. As someone viewing such ratings, I'd entirely skip the latter but would be open to the former if I could infer being part of the in group. The video game equivalent might be a wildly un-optimized title (great if you can run it) vs. a title with subpar gameplay.