Here's my breakdown of the flaws with inference stats like RPM/RAPM:
If it's using Bayesian methods, it's inferring.
E.g., Player A lights up scrubs in garbage time, ergo, his RPM is high, thus there's good "evidence" to hypothesize said performance over 48 minutes against starters.
"Well, it does adjust for opposition via a type of ELO [me: I'm assuming] system."
How can there be any legitimate statistical foundation in this sense when end-of-the-bench players might never play against a starter over the season? Even the sample sizes of bench players against starters will be insufficient to really draw any meaningful conclusions. A bench player might see his starting counterpart for only a few minutes per game. Not to mention the fact that usually in that case, the bench player is fresh while the starter might be near the end of his minute allotment for that particular quarter. So if a player like Bertans comes in and lights up Anthony Davis for three quick 3s and the Spurs increase their lead in that time frame, his RPM (for that particular segment) will be sky high. But it would be foolish to extrapolate that impact over 48 minutes, which RPM does, even when it supposedly "adjusts." It's blind to game conditions and context.
Again, this works fine for something like a drug trial: "These 1000 patients randomly selected from the general population showed a 70% improvement in arthritis after the 30 day trial. From this [Bayesian data crunching ensues], we can safely estimate the efficacy of the drug to be in the range demonstrated during the trial."
But sports aren't a laboratory environment where variables are kept relatively constant. In the drug trial example, every patient had arthritis and took the same drug for the same amount of time. Of course there's variables like age, disease severity, and so on, but at the end of the day, they all interacted with the drug in the same exact way. Kawhi and Kyle Andersen don't "interact" with Lebron James in any similar way. If Anderson does come in to play against Lebron, the conditions will be vastly different from when Kawhi was playing against him. And furthermore, Anderson's sample size will be insufficient to draw any meaningful conclusion about his "real" value. Bench players don't interact with the starters in the same way, and so on.
Again, many of those "variables" do not interact with each other enough to even attempt to "model" anything. It's a highly speculative exercise.
Criticism of Bayes' Theorem:
The last sentence raises the central issue. A sporting event simply isn't controlled enough (i.e. like a lab experiment) to even to begin to develop a complete enough initial hypothesis to work off of. Sample sizes also aren't large enough to effectively evaluate player value. Further issues manifest when you take into consideration that a basketball team is more whole than discreet (i.e whole greater than the sum of its parts). This is why I don't really have a problem with advanced team stats, but I think it's nigh-impossible to tease out individual player value in this context using inference and probabilistic methods. This is why I'm becoming more in favor of "hard" mathematical stats for player evaluation rather than subjective "advanced stats." Even "number stats" like PER are subjective, since it can't really quantify the value of assists.
Now in order to have a solid logical foundation to work from regarding "hard" stats, we need to figure out what wins basketball games (from a mathematical point-of-view). Shooting percentage vis a vis PPG and Usage can be a good indicator. We've seen throughout NBA history that successful teams often have efficient volume scorers (relative to the NBA average).
I also think more value needs to placed on where a player does his scoring. Effective paint scoring has long been a hallmark of great teams, so I think a 25 ppg/50% player who scores 40% of his points in the paint is more valuable than 25 ppg/50% player who scores 40% of his points on mid-range jumpshots. A stat that needs to be created relative this is a "Score off miss" stat. Intuition tells me it's easier to trigger fast breaks off missed jumpshots than it is off missed paint shots. But of course a good team defense can cover up the flaws in the former, which again, speaks to the difficulty of isolating performance in a fluid team sport like basketball.
I also think defense and playmaking will remain largely immeasurable and better evaluated by the eye test.
the only media members that even follow advanced metrics are also the same guys who understand how they work(so they wouldn't view them incorrectly like most internet posters)..
Reply With Quote
