Evaluating Group Play Against the Soccernomics Model

I've spent a number of posts talking about the Soccernomics predictions for World Cup group play outcomes. Now that group play is complete, it's time to evaluate how the model did at predicting outcomes and which teams greatly over- and underachieved versus their Soccernomics predictions.

Ability of the Model to Predict Group Play Match Outcome

One way to judge the accuracy of the model is to look at its overall count of correct and incorrect predictions of outcomes. I have added a third category, "D" for draw, to capture when a push might occur in a situation where a wager is made on a specific match. Figure 1 captures this count.

Figure 1: Count of matches where Soccernomics predicted outcome of win/loss was correct (1), incorrect (-1), and when a draw occurred.

As one can see, the Soccernomics model correctly predicted the winner of 24 group play matches, or 50% of the matches played. The next likely outcome was a draw, with the least likely outcome being an incorrect prediction. This suggests that if wagers were involved in the matches, one could stand to make a good bit of money in wagers without spreads that also utilized pushes.

Of greater interest, however, is under what predictions the Soccernomics model is most accurate. How big does the predicted goal differential need to be to ensure a higher likelihood of predicting the correct outcome of the match? Figure 2 answers this question by using a cumulative distribution function (CDF). A CDF is used to show the likelihood (y-axis) of finding a value equal to or less than the one of interest (x-axis). In this case, I have plotted (1-CDF) to more clearly show the separation of the accuracy of predicted outcomes based upon predicted goal differential.

Figure 2: (1-CDF) for the Soccernomics model's ability to correctly and incorrectly predict a match outcome, as well as a draw.

Two observations can be made about the Soccernomics model's ability to predict outcomes in World Cup group play:

• By a goal differential of 0.5, the model's ability to predict a "not loss" outcome is about 2:1. This gap grows as the goal differential progresses to 1.0. Any predicted goal differential greater than 1.1 means that the superior team will either draw or win the match with virtual certainty.
• The closeness of the "correct" and "draw" lines indicates that the model's accuracy is not good enough to differentiate between a win or a tie. This makes the push all the more critical in a situation involving wagers.

The importance of a win in group play is even higher than expected given the roughly equally likelihood of drawing or winning with such high goal differentials. As SoccerQuantified.com points out, the number of draws in this year's group play was completely normal compared to previous competitions (although an abnormally high percentage came in the first matches).

Performance vs. the Model

Given the model's predictions, I have also quantified which teams have over- and underperformed the most versus the model. One measure of this is the average residual from the model - see Figure 3 for a plot of these average residuals in descending order.

Figure 3: Average residual from Soccernomics model by nation

One can see that four out of the top five teams made the knockout rounds (Slovenia being the exception), and eleven of the fifteen teams with positive average residuals made the round of sixteen. This left only five spots for the remaining teams, meaning teams with a positive goal differential residual to the Soccernomics model were twice as likely to make the knockout rounds than those who did not.

Another way to understand which nations performed most consistently against their Soccernomics prediction is to look at the standard deviation of their residuals to the Soccernomics model. Figure 4 summarizes this data.

Figure 4: Standard deviation of residuals from Soccernomics model by nation

Nations with the smallest standard deviation of performance to the Soccernomics model present a mixed bag of overall performance. New Zealand and Slovenia ranked high on both lists, indicating they consistently outperformed their Soccernomics predictions. Cameroon pretty much conformed to their Soccernomics prediction given their average -0.1 residual and low standard deviation. England and Italy, by ranking low in average goal differential and with a low standard deviation, consistently underperformed against their Soccernomics predictions. Also note that Portugal has the highest standard deviation, which indicates their top spot on the average residual list is likely due to the outlier of the seven goal barrage against North Korea. This means Slovenia's performance was likely the most consistent overperformance in the tournament.

If there is a Tom Thumb award to be handed out, it's a toss up between Slovenia and Uruguay. Slovenia deserves special consideration due to how small and new their country is. They have only two million people, with a much smaller population base of 20-30 year olds that could play in the World Cup. Uruguay deserves consideration for their high score on the average differential scale, and their win in the Round of 16. To win that match they had to overcome both the Soccernomics prediction and Footballer-Rating.com prediction that both favored Japan. They will face a challenge similar to what the US faced in the first round - a Soccernomics prediction that favors them but a Footballer prediction that favors Ghana. If they make it through that match into the semi-finals, they will truly be a Tom Thumb winner. Until then, Slovenia's high average residual and low standard deviation make them the winner.

Notes About the Round of 16

As I stated here, the Soccernomics model predicted smaller goal differentials for the Round of 16 - only one match had a projected goal differential higher than 0.6. In fact, the one game that did have a projected goal differential higher than 0.6 ended up being an incorrect prediction, although it did fall into the second most likely outcome for regulation time score (a 1-1 draw) based upon group play matches with a predicted goal differential of 1.0 or more. On the other hand, the Footballer-Rating.com method of looking at group play performance of the two teams to determine likely match outcome predicted the correct result in seven of the eight matches (Paraguay's win against Japan was the one outlier). This model seems to be a much better predictor of outcome in the early stages of the knock out round. It will be interesting to see if this trend holds. The statistics team at Footballer-Rating.com does not seem to be updating team performances based upon knockout round matches, which will force me to rely upon the group play data. I would have preferred to do a three match running average to capture the effects of playing against tougher opposition as the competition goes on.