The negative reactions to Shanahan’s perspective on “analytics” are great examples of how worshipping at the altar of empirics can rot your brain.
Decisionmaking in football is an infinitely complex and intricate web of interconnected components. Historical trends are informative, but in a game scenario where you know he knows you know—and you have unique matchups/reads on situation-specific tendencies, so subtle that any kind of aggregate statistic would completely erode them away—stubbornly conforming to the guidance of a posteriori generic historical tendencies is just stupid. Especially when the supposed edge is only 1% over random chance.
That’s not to say purely model-derived decisions are bad, but you have to understand the limitations of the model you’re working with and what value its conclusions actually have in real-world scenarios. Especially in complex games, not every variable can be modeled; in that sense, unequivocal deference to model-derived recommendations based solely on the fact that they came out of a mechanical system is simply naïve.
That kind of Blind Number Worship is no different to any other breed of credentialism; people make up their minds that information from a specific source is gospel and turn off their brains—mindlessly accepting anything from The Source and vehemently rejecting anything that contradicts it.
You see just how potent that effect is with these people concluding with unquestioned confidence that Kyle Shanahan—a highly successful coach with a decade of head coaching experience at the highest level and 2 Super Bowl appearances—is somehow “clearly wrong” in an extremely marginal situation while internet randoms with no accolades or even relationship to the game at all are “clearly right.”
Shanahan doesn’t technically characterize it the correct way—talking about limited trials as the motivating factor vs limitations in the prescience of modeled outcomes vs his own intuition—but I have to imagine he actually means his instincts allow him to err towards higher success probability within the margin of error for that one decision. Especially given the example he’s using, with 51% vs 98% model-derived guidance.
Kyle Shanahan says a 51% analytics recommendation won’t convince him to go for it because he gets one shot, not an hour at a blackjack table
“I don’t want to hear about it if it’s 51% of the time we should go for it.”
“As I always say, I’m not sitting here playing Blackjack for an hour with no one else at the table, and eventually I’ll get 51% of it. Like, I have this one try, and I have a feel for it. I’ve been doing it too long.”
“But what I want to know from him, when it’s not a feel, when it, no, this is 98%. We don’t even need to talk about it. So we don’t need to talk about it. Tell me when it’s that, and I’ll just do it.”