Numbers and Names (The Fallacy in Data and Name Recognition)
Vermanubis - May 22, 2017I regularly post these and things like these on my Twitter, as well as other Smash stuff: twitter.com/Vermanubis
Because there’s ultimately no concrete way to delineate between players of similar skill levels, we rely almost exclusively on data to inform our opinions on relative skill.
While we’re right to do this for lack of any other practical metric of assessment, most methods we use to gather data and stratify players, if not obviously flawed, are still flawed, but in much more subtle ways. The two major means of assessment we use are tournament placing and people beaten. I want to explain why both are necessary, but really only approximate actuality rather than reflect it with 100% fidelity.
Tournament Placement
This one seems fairly cut and dry: you place higher than someone else, you obviously performed better. Moreover, if you placed high, you obviously did well!
I… don’t know about that.
You see, while those may be true as a general rule, they’re not necessarily true. As I said, this form of measurement only approximates actuality as a matter of likelihood; there’s nothing about a high tournament placing that necessarily entails the statement two paragraphs ago. In other words, it is likely that those two statements are true, but they are not true all the time. This means that we should be skeptical of these figures out of context.
Consider I enter a major and I place 250/1000. That appears to be a severely mediocre placing that’s going to send me home with a sore tailpipe. But what if I told you I was seeded dead last and had to beat Abadango, Ranai, and VoiD to get there? What if one of them was simply upset by someone else and I had to face them early in bracket?
250 doesn’t seem so bad now. Now what if I told you I placed 1/1000? Woah, far out, man!
Haha, not so fast, Eager McBeaver! Turns out everyone but me in the tournament was either a small child, common housefly or a can of delicious baked beans! Minus beating the can of delicious baked beans, this is decidedly unimpressive, and I hope all you folks without profound mental deficits would agree!
To cast this more concretely, consider this post by Kennicky from Reddit:
"I think a really good example is HugS at DH Austin (I know it's the wrong game but the point still stands)
The reality is that funky shit happening in bracket can dramatically lighten or encumber someone’s run. Beyond this one, we’re hardly starved for an example of seeding being subverted by an upset or two, resulting in remarkably caddywampus -- as far as projections go -- results. I personally have had placings where I had to beat 6+ PRed players in a row that were numerically inferior to tournaments with far easier brackets.
People Beaten
This is the genuinely tricky one to navigate because of our intuitions. It’s sort of a reaction we have to double take when we see someone take a name we recognize out. It’s exciting! The drama is infectious! In spite of this, though, we still have reason to be skeptical. Why?
Again, the name of a well-known player is only an approximation of what we’re trying to measure. It’s assumed that whatever that player did in an upset match they lost was faithful to their standard, almost as if it doesn’t matter how they played so much as their name.
This is an idea that is vastly underappreciated. The name alone of the person we’re watching can bend us between assessing the exact same match as utter shite or the supreme echelon of gameplay to which we all aspire; we all, to some degree, have a player bias. The truth remains though: top players can still err, play off kilter, be ignorant to a particular MU, or otherwise perform in ways we’d not recognize them if we didn’t already know it was them. If our honest pursuit is that of excellent gameplay and the accurate stratification of skill levels, then we should certainly prefer an excellent performance that is lacking the glory and face value of name recognition to an underwhelming performance by a celebrated player. If Ally starts having an aneurysm mid-set and plays in a way that reflects that, him simply being Ally doesn't turn your win into something more than it actually was. Similarly, if an unknown player gives a stellar performance, it should be treated as such and not devalued for lack of name recognition. If I play two people, Zero and an unknown player, and Zero plays like hell and the unknown player plays marvelously, then Zero being Zero should not magically make my win over him more meaningful than over the unknown player if he played like hell.
(Reminder: this is with respect to an individual level; not on a large-scale one where things like PGR rankings are calculated. As I mentioned in the beginning and will mention again at the end, I'm discussing actuality, not rankings. For the purposes of rankings, this is all highly impractical)
Simply put, we lose sight of what we’re actually trying to measure with all these approximations and indicators such as placings, names and so on: excellence or lack thereof. That’s why I encourage people so frequently to be skeptical of these indicators in a vacuum, and instead to prefer, above all else, the direct, observable quality of gameplay irrespective of name recognition and numbers. These things are all very necessary for the sake of practicality, but should be treated with healthy skepticism on an individual level instead of dismissing high-quality gameplay for lack of name recognition and, similarly, extolling what might be mediocre gameplay simply because we make assumptions about numbers and names.
If this post did not give you some kind of exotic disease of the mind such that you might even consider reading another in the future, I post updates as well as other Smash "musings" to my Twitter: twitter.com/Vermanubis
This blog post was written by a SSB World community member. Share your Smash 4 knowledge by creating your own blog post now.
Comments
Gleed - May 23, 2017To the first paragraph, you're right, it doesn't provide a solution; primarily because I don't think there is one, at least, not a practical one. My goal here wasn't necessarily to remedy the problem, but to express what I thought were two underappreciated shortcomings in our main assessment methods.
To the second, it's not that people should feel that their wins/losses are more or less valuable than they actually are so much as encouraging a focus less on /indicators/ of skill and more on the raw demonstration thereof.
Smoke is an index of fire, but isn't fire itself. Likewise, a well-known name is an index of high-quality play, but does not itself guarantee that high-quality play will be there (see: someone playing hungover or something).
As for the Dabuz thing, I replaced it because it was definitely starting to hang some people up and distract from the point. And you're right about it undermining the point a bit, though my intent was to give an intuitive, easy-to-digest example rather than switch hands for the sake of an example =P
Lastly, totally agreed that the PGR overall does a good job. Like I said above, I by no means think there is a practical solution to all of this; after all, I mentioned at the beginning that there's no concrete way to do so, hence why we rightfully rely on averaged-out data.
The point I just wanted to really underline was that, irrespective of practicality, ultimately, name recognition and bracket placement are /just/ indicators. Highly accurate or horribly inaccurate, they themselves are indices of high-quality gameplay, but don't necessarily entail it, and I think it's important on an individual level to understand that.
Vermanubis - May 23, 2017For that reason, I don't think it's fair to judge only "the direct, observable quality of gameplay" for the purpose of evaluating skill. An oppressive player ranked #1 on a local PR can be made to look clueless by a PGR player, but this does not invalidate the clear skill that the player has.
The relative nature of skill and ability is what makes skill so impossible to quantify.
duncathan - May 23, 2017You must log in to comment.
These observations, while they address flaws in the system, don't really provide a great solution or alternative. I think that measuring skill based on "direct, observable quality of gameplay" is valuable, but what good does quality gameplay do for you if you aren't winning just once, but consistently, at the end of the day? I learn a lot by watching matches between two excellent players, no matter the outcome, but in the end if I had to say who the better player is, it'd be the one who is winning more often.
I suppose I'd just be cautious because as I read this post, it seems to encourage a more destructive attitude than a constructive attitude. I can see why this mindset might help someone to feel more valuable when they aren't necessarily getting strong results (whatever that means) or taking wins of of "big names." However, I think this treads the line with the following example, one we're all too familiar with (particularly those of us who are low-tier mains): "You beat X player in bracket? He must have had an off day." I don't think your post is completely in line that extreme, but I think it can come off that way, particularly when you "encourage people so frequently to be skeptical of these indicators in a vacuum" and to place a higher premium on the aforementioned observation of gameplay. And I think it is somewhat contradictory to justify a mindset skeptic of wins over ranked players in the same breath as devaluing Dabuz' win at Civil War over players not ranked very high by numerical measures. (Though I do agree it casts a shadow on his win.)
I suppose when people I know get any kind of results, particularly against great players, I get excited because I think it wasn't just the "better" player's loss, but also the underdog's victory.
I think, generally, that PR's and PGR do a decent job of weighing player skill and activity in that they measure more the *consistency* of a player, rather than outlier performances, though they may create an artificial emphasis on players who are most active. (Which is fair--if you want to be considered great, then you ought to play.)
I think it's interesting to consider smash results with an analogy to something like sales performance. The most valuable salesperson to a company is the one who brings in the most money every month, not the one who lands a large client from time to time with little to show in between (as exciting as winning these large clients would be). There is something to say for the others, though--those great salespeople who bring in a respectable amount of money, whether or not they win over the most important prospects every time. However, no company would do well to promote their salepersons who have the best presentation skills, the most persuasive arguments, or the most education when they don't have the sales to back it up.
Again, these are just some of the thoughts I had as I read, and I'm sure I've misinterpreted it in some ways. Good read, nonetheless, and I'm sorry my comment turned into kind of an essay. Interesting things to think about.