Hard-hitting Data: Why Metrics Matter

I reviewed video footage of a student playing a PLO session recently and a seemingly innocuous hand caught my eye. The student had raised the betting pre-flop and been called by a player in the big blind. The flop came down:

7♠7♦2♣

His opponent checked, my student bet and his opponent raised. My student raised the betting again, his opponent raised the betting once more in response and my student folded his bluff. From my student’s perspective that was all there was to it.

I have omitted one piece of information that I was privy to as the viewer: my student had raised his opponent’s check-raise almost instantaneously. The problem with this timing is that if he had a strong hand here our hero would need to have thought a little about his decision before raising. Our hero had a bet timing ‘tell’: he had unintentionally emitted information which a skilled opponent could use against him.

It is likely that our opponent in the example above recognized my student’s tell. If so, he was using a superior decision metric to outplay my student

Poker players draw conclusions about our opponent’s hand ranges based on information that they don’t explicitly tell us. This attribute differentiates poker from other games; chess is predominantly a game of calculation, poker is predominantly a game of inference. The ability to use inferential thinking to make decisions is a highly developed skill-set among poker professionals with even greater utility in the ‘real world’. In this post I introduced a four quadrant model of information transfer which I include again for the reader below:

This diagram puts the type of information we handle as poker players into context; we are especially interested in the ‘Third Quadrant’ of Impressions- understanding information emitted unintentionally.

As a reminder, the question this blog seeks to answer is:

“How do we build a successful approach towards our goals in a world shrouded in uncertainty, when we don’t even know the path?”

Historically this has been an exceedingly difficult question to answer since gathering data was a slow and expensive process. Yet now access to an abundance of data which was hitherto either impractical or impossible to collect and store has shifted a vast untapped library of information from the fourth to the third quadrant. It is in this Third Information Quadrant that my three apparently distinct areas of interest: poker, self-optimization and the games of society all reside.

The rewards for the development of data collection tools have been great for the first internet generation, but there are far greater rewards to come for those who can best navigate these new oceans of data. Building metrics to interpret this data is essential for improving our inferential thinking and a valuable tool for the Meta-strategist.

Every time you make a decision you use a metric, although you may not be consciously aware of the metric which you are using. It’s important to recognize that, barring the ‘free will’ caveat, all decisions involve the use of data, although the data used does not have to be relevant to the decision in question!² Even if you decide to simply flip a coin to make a decision you are still generating data for a 1-D metric.

It’s easy, and intellectually lazy, to abdicate responsibility for the more challenging decisions we face in life. Most people console themselves with the idea that some ‘expert’- in the 18th century a priest, in the 21st century a scientist- will solve the problem for them. But the challenge of making decisions based on incomplete information cannot be magicked away with a scientific study. My favorite example of the perils of relying on scientific studies can be found in a popular weight-lifting blog,

“Alwyn Cosgrove once said that if you took a lifter and had him perform a 1 Rep Max in the bench press, and then later the same day had him perform a full bench press workout complete with assistance lifts followed by having him test his 1 Rep Max again, the said lifter would test lower. Thus, you’ve just ‘proven’ that weight training makes you weaker.”

The error committed by our hypothetical researcher above is one of selecting an inappropriate reference class from which to draw conclusions. The mechanism by which weight training improves strength takes place over a time window larger than that used in the proposed study. Thus this study would not yield useful empirical data by which to test a hypothesis of the efficacy of weight training for improving strength.

Fortunately we know enough about exercise physiology to recognize the flaw in this study, but it’s hard to avoid such flaws in fields where we have less knowledge. We’ll investigate reference class problems in greater detail in subsequent articles but for now this example serves as a warning against delegating your decision-making agency to an ‘expert’.

Now that we’ve established that metrics matter, let’s get on with the business of using them.

There are three stages in using metrics to interpret an incomplete data set:

1. Situation Recognition: Before we can use any metric we need to recognize the specific situation as belonging to some broader, reference class of situations which have sufficiently similar features. A reference class which is too broad will conflate unrelated situations and lead to inaccurate decisions. A reference class which is too narrow will occur too infrequently for building a metric to be worth our time. Notice that any one situation can be a member of infinite reference classes and the appropriate reference class depends on the objective of the observer. In poker games we have the distinct advantage of caring about only one payoff: the Expected Value of the strategy.
2. Factor Selection: We are going to build metrics with a finite number of factors. To do so effectively, we would ideally be aware of all relevant factors and then select the most important to include in our metric. In practice we often don’t know what all the relevant factors are, which is why factor selection is a test-driven process.
3. Processing: Finally we need to combine the factors in a way that consistently yields decisions consistent with our objectives. In order to do so we need to understand the structure of the game we are playing. This is where analytic methods are invaluable.

These three stages are strictly hierarchical, in the sense that no level of skill in a latter stage is high enough to compensate for an error at an earlier stage. An excellent metric is rendered useless when we try to fit it to a situation outside of the reference class within which it is effective. Superior processing power is mere smoke and mirrors when we have omitted a dominant factor.

The examples chosen in this article serve to illustrate the hierarchical nature of metric-based methods. Our weight-training anecdote demonstrated that Situation Recognition dominates Factor Selection. A researcher could be collecting data on all of the most relevant parameters yet if he collects this data from the wrong situations his effort is worse than wasted; it is misleading.

The poker example in the introduction demonstrated that Factor Selection dominates Processing. Notice that a game theory based analysis of that situation as approached by arithmeticians³ would be ineffective were the parameter ‘bet-timing’ not included in their model. Their bet sizing and frequency with each part of their range could be ‘perfect’ and so apparently indistinguishable to an observer looking for sizing and frequency tells.

Yet the arithmetician’s range would not be balanced from a bet-timing perspective since he had not included ‘timing’ as a parameter in his model. The game is bigger than the arithmetician realizes; he has used an inappropriate reference class and his model will fail, because his metric does not allow him to interpret what he’s reading.

The next article in this series will introduce decision-making heuristics. We’ll discuss how heuristics can be used to rapidly generate metrics and how to integrate heuristic methods with game theory.

QUESTION(S) OF THE WEEK: Can you think of a time when you have used a metric that was inappropriate for the situation? How did you discover your mistake and what did you learn? Please let me know in the comments.