MLS 24 Under 24 : The Hopelessly Futile Challenge

I also wrote a lot of words about my methodology and criticism of the process below.


The Challenge

I have been a little critical of the "MLS 24 Under 24." It is still so early in the player's careers and so much can change.  For evidence, look no further than last year's winner Darren Mattocks, who has fallen completely out of favor with Vancouver.  Naturally, as one does, I voiced my skepticism over Twitter:

"24 under 24 is like looking at a "power ranking" 5 games into the year. Fun, but hopelessly futile."

 Andrew Wiebe of mlssoccer.com responded and offered to send me a ballot. "Hopelessly futile" challenge accepted.

I am a little familiar with the ballot after seeing Matt Tomaszewicz aka "The Shin Guardian" post his ballot on his website. Finding a consistent approach to grading players is the hardest thing.  Contrary to its intention, I actually think the rubric makes the process harder rather than easier (see Criticism).  There are five categories you must grade each player on from 1-20 (why 1-20?): Technical, Tactical, Physical, Personality, Potential.  Not wanting to get bogged down with this process just yet, I instead "force ranked" my top 24 and, only after doing this, assigned point totals.  I assume the majority of media members who filled this out did the same.

Methodology

In my opinion, the two questions that really matter in this exercise are "how good is this player now?" and "how good could this player become?" I am a stats guy so, to answer the question of how good each player is now, I started with each player's statistical profile, courtesy of whoscored.com and squawka.com.  Context is key when looking at statistics, so every player's profile is compared against: other players at their position around the league, other players at their position in 24 Under 24, other players at their position on their own team.

Figuring out the potential of each player is obviously a much more subjective process.  Most players on the list I have only seen play 5-15 times,  limiting my ability to pass any sort of conclusive judgement.  Large factors in determining potential are a player's physical capability and their age/how much time they have spent in MLS.

Once force-ranked, coming up with the point total for each player is fairly difficult.  The Personality category immediately jumps out as problematic: who cares if you're a nice guy or a jerk, as long as you play well and make your team better? Further, while I have biases and might be able to guess what type of personality a player has, I really do not know them well enough to have an informed opinion.  I am sure this goes for most media members as well.  I basically punted and gave everyone a "10".

To further simplify matters, I took Technical and Tactical to represent how good the player is now and Physical and Potential to represent how good this player could become.  As such, a player's score was the same across each set of categories.

Criticism

The grading rubric is unnecessarily complex and, in my opinion, distorts the voting process.  A player's personality counts just as much as their technical ability or potential?  It does not make sense.  But the real problem is the variance in point totals.  The range in my point total was 14 (between 74 and 60) across 24 players, less than a point difference between each ranking.  Looking at the only other public ballot (that I could find), Matt Tomaszewicz had the exact same range of 14 (between 72 and 58), though he somehow scored 25 players (?).

The problem is best exemplified by a player like Shane O'Neill, who was ranked #3 by Matt and just outside the 24 by me.  Under the current methodology, combining our two ballots, O'Neill would actually slide all the way to #20.  Harsh.  The problem with the current methodology is that instead of measuring variance of opinion, it ends up measuring variance of methodology.  The easy solution is to do a more traditional (i.e. Heisman, MVP, etc.) style where #1 gets 24 points, #2 gets 23 points, #3 gets 22 points.... #24 gets 1 point.