An Indictment of the MLS Playoff Structure, Part 4: Quantifying the Impact of Game Differential on First Round Odds

In my last post I used binary logistic regression (BLR) to show the impact of various EPL match attributes on the likelihood of winning a match.  Soon after beginning the work I thought of another good use for a BLR model - identifying the factors that truly matter in predicting (as compared to retrospectively assessing) first round MLS playoff success.  Regular readers know I am no fan of MLS's playoff system that seems to buck the trend of most leagues using a top-of-the-table format to determine their champion.  I took a pass at explaining the phenomenon of teams with fewer games played winning a disproportionate share of first round series since 2003, but at the time could not prove that it was one of the few statistically significant predictors from the many identified by Climbing the Ladder (CTL).  Since then I have used CTL's lineup database to construct a BLR model of the major factors identified by CTL and isolated the statistically significant factors.  The results suggest that MLS has a lot of changes to make to continue to improve their playoff format they seem to love so much.

The Data Set and a Few Summary Statistics

I utilized CTL's lineup database to create the following statistics:

• Difference in the number of games played
• Overall goal difference
• Difference in coach experience (MLS games only)
• Difference in seeds
• Home record difference
• Away record difference
• Regular season goal differential between the two teams

Here are a few interesting summary statistics regarding several of the attributes above for the 2003 through 2010 seasons:

• Games played difference: The maximum difference came in the 2008 NY/Columbus series where the Crew played 14 more games than the Red Bulls.  This was due to the Crew playing in both the old and new format for CONCACAF Champions League (CCL).  The median differential is 2 games.
• Overall goal difference: 2005 through 2007 saw some of the highest overall goal differentials between first round playoff participants - one series each with 22 and 23 goals, and two series with 27.  The median goal differential was 9 goals.
• Difference in coach experience: Sigi Schmid is the king of mismatches in manager experience with the largest gap of 196 games realized when his 2008 Columbus Crew defeated the Kansas City Wizards.  The median difference is 68 matches.
• Regular season goal differential between the two teams: The peak of regular season goal differential between two teams was witnessed in the 2006 through 2008 seasons, when an unbalanced schedule saw teams play each other up to four times during the regular season.  Chicago's 2008 first round series versus New England witnessed the largest such goal differential - 8 from four regular season games.  Chicago went on to win that first round series, while the median value for the first round series is 2. 

The Results of the BLR

After compiling the data for every first round playoff series from 2003 through 2010, a BLR was constructed to predict the likelihood of a team winning the series.  Dummy variables for each year were constructed to ensure no special causes were missed.  Terms having a p-value greater than 0.05 were successfully eliminated until only two terms remained: manager experience differential and games played differential.  All other terms - overall goal difference, difference in seeds, home record difference, away record difference, and regular season goal difference - were not significant by a mile (most p-values were equal to or greater than 0.40).

Plots of the the changing odds with various manager experience and game differentials is shown below.

The plot of manager experience differential is counter intuitive, but is born out by the statistics.  Apparently having a less experienced manager actually bodes well for a team in the first round.  Perhaps it is that long term managers have more data points in the MLS playoffs, increasing the likelihood that the random nature of the playoffs will lead to their increased number of losses.  Perhaps newer, less experienced managers can take a team by surprise, or are managing teams that do not perform well the prior season and thus align with the teams with fewer matches.  Whatever the cause, it seems that fewer games for a manager, relative to the opposition team's manager, bodes well for the team with the less experienced manager.

More interesting is the intuitive relationship between the game played differential and the likelihood of winning the first round playoff series.  The equation provides a very clear relationship between the game played differential and resultant odds.  The relationship is largely linear from -5 to 5, meaning that it's about a 7.5% change in odds for each incremental game difference.  As game differential approaches more than five matches, the incremental benefit of increased or reduced game differential is minimal.

What does this mean for the 2011 playoffs?  The most direct assessment is that MLS missed a golden opportunity to correct this imbalance by not going to a two game series in the first round of the playoffs.  I commented here how I would have liked to have seen such a re-balancing prior to MLS's announcement of the 2011 format, as well as my reaction to the announcement that the first round would be a single match while the second round will have the usual two-match series.  If history is any indication, MLS essentially gave the lower seeds a 7.5% advantage in odds of advancing to the conference final by not making them play an extra game in the opening round.

We'll see how things play out in the 2011 playoffs and will update this study once they're complete, but I am not holding out hope.  With my Sounders having one of the league's more experience managers and their desire to go deep in the US Open Cup and CCL, I think it will be another year without an MLS Cup.

Perhaps MLS really does like this parity that borders on complete unpredictability.  Allowing MLS clubs to buy championships like Chelsea is not what I want, but the fact that no prior rational metric correlates to first round playoff success suggests MLS has some major adjustments to make.  It seems as if MLS has swung the pendulum so far to the side of parity that no club's supporters know what to expect from regular to post-season, let alone year-in and year-out.  Ultimately, this holds back the professional game's success and growth in this country.

A Few Quick Reactions to the 2011 MLS Playoff Format

MLS has finally released their 2011 playoff format, and I certainly have a few ideas based upon my series I did on their past playoff formats and outcomes.  I dealt with my recommendations for the 2011 format in this post.

1. The format largely follows what I outlined in my "Recommendations for 2011" in this post.  The top six get a bye, the Supporter's Shield winner gets the lowest seeded team out of the wild card round, and we generally eliminate all the crossing over from conferences found in the last few seasons.
2. What I don't understand is why the wild card round is one match, then the conference semifinals are two legged, and then the conference finals and MLS Cup are back to single matches.  This makes no sense, and it just seems like MLS is trying to follow previous playoff formats in some desperate attempt the provide continuity.  A much better, and natural, approach would have made the wild card round a two legged series of matches.  This would not only provide a natural progression via a reduced number of matches each round, but it would also add two critical matches to the lower seeds and close the likely gap in matches played between them and their second round competition.
3. Yet again, the Supporter's Shield winner gets minimal benefit.  If the two-legged series was played in in the wild card round, the Supporter's Shield winner would be guaranteed to play all their playoff games at home and benefit from MLS's outstanding home field advantage.
4. MLS didn't announce the location for the MLS Cup yet, so we'll reserve judgement on my recommendation to allow the highest seeded team of the final two to host it.
5. MLS needs to stop making excuses for this awful playoff format.  The story on their website is chock full of excuses from the league.  Here are a few samples:

"It will mean that the battle and the race for the playoffs will have added intensity and last longer through the regular season. And that, on balance, should be a good thing for the play on the field, and for the fans in the stands and at home."

This statement defies logic.  Letting in the highest proportion of teams to the playoffs in over five years won't enhance competition, it will dilute it.  You can't have the lowest standard of admission, with the greatest number of teams admitted in the history of the league, and think that it won't bring more parity and a lower standard of playoff team as a result.

“There are so many variables that any system is imperfect in that regard,” Rodriguez said. “You could be facing the lowest-remaining seed who’s on a 10-game winning streak to close the season and they’re the hottest team in the league. … Any playoff system has, within it, some fault or a flaw that could easily be pointed to.”

Not if you followed my recommendations and kept the two-legged series in the Wild Card round!  MLS is seriously trying to blame "the random nature of the playoffs" to excuse their failure to properly design a playoff system over which they have total control.

“At the end of the day, you try to create a playoff system that is fair and relatable to the regular season,”

This is only the case if MLS's goal is the complete random nature of a league that exhibits the most parity of any US pro league (too much parity, in my opinion).  Perhaps it's that parity and the random results it produced that led to the awful MLS Cup ratings last year?

I can only hope MLS is serious in their desire to "spend the next bit of time trying to finalize a long-term playoff situation that can take us into the future."  We deserve a better system for determining the MLS champion.

An Indictment of the MLS Playoff Structure, Part 3: Making the Best of a Bad Situation

Note: This the third and final post in a several part series that I have written about MLS's playoff structure and how it penalizes successful teams.

In my first two posts in this series I highlighted the effects of letting too many teams into the MLS playoffs and the resultant penalty good teams pay in the playoffs by playing a greater number of games during the regular season via domestic and international competitions.  In this post I will focus on solutions to these problems.  I use the byline "Making the Best of a Bad Situation" in this blog post's title because we all know that Don Garber and MLS will not ditch a playoff format - they are (mistakenly) wedded to the idea that US soccer fans want a league that is eventually the size of the NFL and requires a playoff format to determine its champion.  So, after listing my ideal solution, I will spend time exploring how the 2011 playoffs could be formatted to make the best of the already announced 10 team playoff field.  I will then move on to how future seasons might benefit from a reduced number of playoff teams and modified formats.

The Ideal Solution

Before I delve into making the best of the poor hand dealt to us by MLS, I would like to lay out what I envision would be the ideal format for determining MLS championships.

1. Limit MLS to 20 teams: This preserves the ability to run a balanced schedule.  Perhaps I am writing from a bit of personal interest, but a balanced home-and-away schedule is the most fair to the teams and is great at providing statistical insight.  Each team gets to play every other one twice, and home pitch advantage is effectively wiped out.  Such a format is critical in delivering my second recommendation.
2. Play a top-of-the-table championship format: After 38 matches, we would truly understand who was the best overall team throughout the season.  No upsets allowing eighth seeded teams to knock out the number one seed.  No poor form for half to two-thirds of season that is bailed out by a hot run at the end of the season.  There's a reason the rest of the world runs such a  format - it provides stability to the teams and supporters and justly rewards good long term performance.
3. Reform CCL qualification: If (2) is realized, the US Open Cup then becomes the major domestic knock-out tournament just like most other national tournaments.  Make the US Open Cup championship worth something - make it the automatically qualifier for CCL group play along with the MLS top-of-the-table champions.  To add drama and competition to the MLS single table format, make the second place team the play-in for CCL.  Overall, this would provide greater incentive for winning the US Open Cup and ensure our CCL teams truly are the best MLS has to offer.
4. Implement promotion and relegation: This system works in nearly every other soccer nation.  For those who worry about the health of MLS and soccer in the US, they should realize that attendance at matches in the US rival those found in top leagues outside of England, Germany, and Spain.  What is holding back growth in the US is MLS's obscene franchise system and associated fees.  Such high barriers to entry stifle the development of the lower tiers of the US soccer pyramid, and are not found in other parts of the world.  When soccer rich cities like St. Louis can't field a second tier professional team and the entire second tier is at risk of losing its sanction because of financial difficulties, it's clear something has to change.  Simply put, the closed market, single entity concept is holding back the development of the professional game and our national team.  It's a hurdle not found in other countries, where teams who work their butts off at lower levels earn the just reward of promotion without having to pay the cartel and extra entry fee.

I am not delusional.  As I said at the outset, I think this proposal would never be implemented.  The main reason is that it runs counter to every fiber of American professional sports, which are all about the league and guaranteed revenue and not the team.  Team success (both on the field and financially) is only allowed in so much as it builds the league's product.  The league redistributes the earnings from team merchandise, stadium revenue, and TV revenue - more so than they do in soccer leagues in places like Europe.  If the league doesn't like the deal one of its owners is getting in a town, they support uprooting the team and moving them to another state or a completely different section of the country - we're talking a much bigger relocation than Tottenham's potential move to the Olympic Stadium (and resultant supporter resistance).  In the United Stats it's about moving a basketball team from Seattle to Oklahoma City or a soccer team from San Jose to Houston.  As is often the case, such moves are condoned by the league because local tax payers are unwilling to be held hostage to build a new stadium that the teams are unwilling to finance themselves (precisely because the stadium is more expensive than the team can justify paying for themselves).  Moving towards a promotion/relegation single table format that rewarded consistent performance would encourage reasonable growth at multiple levels of the professional game, improve the quality of our national team by developing more talent, and avoid the boom bust cycles of the original NASL and first decade of MLS's

So if the ideal solution is not a realistic one, how can we make the best of a bad single entity playoff solution?  How can a full season's effort be rewarded come playoff time?  The answer is two fold - one solution for the 2011 season that will see 10 playoff teams, and another for subsequent seasons where the number of playoff teams is still undetermined.

Recommendations for 2011

Given that MLS has indicated they will break 10 of the 19 teams to the playoffs for the 2011 season, how can the as-yet-undefined format be defined so as to reward regular season play and not make it an afterthought as it currently is regarded?  Here are a few suggestions.

1. Give the top six finishers a bye:  If we want to encourage teams to compete for playoff position from top to bottom, we need to give them a just reward for their successful efforts.  Currently, there is no real advantage to being a top seed.  Giving seeds 1 through 6 a bye, which would effectively give them two weeks between their final regular season match and their first playoff match, would give them much needed rest after a grueling regular season, US Open Cup, and CCL group play.
2. Make seeds 7 through 10 play a two-leg first round: Making the lower seeds play two extra games will help level the playing field when it comes to game differential.  Such an approach would have moved 5 of the 12 series from 2003-2010 that had a game differential of four or more to series with a game differential of three or less - crossing to the right side of the magical four game differential I highlighted in my last post in this series.  I'd also make them play it on a weekend/weekday swing, which would essentially make the winner of the series play three games in seven days when the second round match was factored in.  Meanwhile, the team they faced would be coming off two weeks rest.
3. Make away goals count for more than home goals in the first round: Nearly every other two-leg knockout round competition uses this rule.  For some unexplainable reason, MLS does not.  Such a change would reward risk taking by away sides, and recognize the inherent difficulty in scoring goals away from home.  It would also likely help minimize the number of opening round matches that went to overtime and PK's in the second match.
4. Re-seed after every round: Implicit in this suggestion is the request to forget about conferences once the playoffs start.  The reward for finishing with the best overall point total (or tie breaker) should be that that the highest seeded team host the "weakest" team each round.  Higher seeded teams shouldn't be forced to travel or endure any hardships come playoff time if they've worked their rears off for 34+ matches.  This will also help to eliminate MLS's current crossover system that has the propensity to hurt top performers.  Case in point: in 2010 the Western Conference took six of the available eight playoff spots, yet the bottom two teams crossed over to play the two Eastern Conference teams, both of which had lower point totals than the top two teams in the Western Conference
5. Take advantage of MLS's superior home field advantage:  As Scorecasting has shown, MLS teams have the greatest home field advantage of any US sports league or major soccer league world wide.  Let's put that to good use and provide greater rewards to superior regular season performance.  MLS should not only re-seed the matchups every round, but they should also have the higher of the two MLS Cup participants host the championship match.  Therefore, there truly is a benefit to winning the Supporters' Shield - controlling one's own destiny as to where matches are played all the way through the playoffs.

Announcement of the 2011 playoff format is expected any day now.  We'll see if any of the changes conform to the suggestions above.

A New Playoff Format for 2012 and Beyond

By 2012 MLS will have its 19th franchise (Montreal), unbalanced conferences, and possibly an unbalanced schedule again.  Everyone fully expects the league to expand further, not stopping at the 20 teams found in most other domestic leagues.  How should their playoff format change to match these changes and improve the quality of the playoffs?

1. Run a single table, and break only the top six teams to the playoffs: With Montreal in the league, there will be no way to balance the conferences.  This would be a perfect opportunity for the league to convert to a single table.  Limiting the playoffs to the top six will also greatly improve the quality of teams that make it in.  The regression equations developed in the first post in this series indicate a team in the top six would need to earn 51.5% of the available points and have a goal differential of nearly 8 - both are massive improvements over the numbers required since 2005.  It would also lower the percentage of teams that make the playoffs to 31%, making it the second lowest percentage of the US professional leagues profiled in my first post.
2. Use the same suggested play-in format from 2011 for seeds 3 through 6:  Again, it will cut down on game differential in the second round, provide greater incentive for finishing in the top two via an extra week of rest, and will ensure away goals count for more.
3. Re-seed for the semifinals just like 2011, and have the higher seed host MLS Cup: Give good teams every advantage they deserve for sustained quality throughout the regular season.
4. Reform CCL qualification along the lines of (3) in the "Ideal Solution" section of this post.

We're at least a year away from Commissioner Garber announcing anything about the 2012 playoff format.  I suspect he will resist any attempt to go to a single table, especially when such a format becomes difficult to maintain given the geographic reality of the league and his continued desire to expand beyond twenty teams.  Nonetheless, following the rest of my suggestions would greatly improve the quality of teams that make the playoffs.

Conclusions

It's clear that MLS is intent on keeping a playoff format to determine their champion, as they mistakenly believe that US soccer fans and fans of other sports that they are trying to attract want an Americanized version of the world's game.  If this is the case, there is a right way and a wrong way to implement a playoff system.  Right now, MLS is going about it the wrong way.  They are letting way too many teams into the playoffs, and the resultant chaos turns fans off as there seems to be absolutely zero reward for an entire season of hard work.  The wide open playoff format, combined with commitments outside of MLS, has produced little predictability from one season to the next.  If MLS is to insist on having a playoff format, it is my hope that they adopt a number of the proposals made above.  Restricting the number of overall participants, and forcing lower seeds to play more games in the playoffs, will improve the chances of higher seeds making it through to the MLS Cup and hopefully provide some predictability to the current chaos that is the MLS Playoffs.

An Indictment of the MLS Playoff Structure, Part 2: Too Many Games for the Good Teams

Note: This is the second post of a several part series that I will be writing about MLS’ playoff structure and how it penalizes successful teams. Part one is located here.

In my first post in this series, I explained how MLS’s current playoff structure and generous standards to determine which teams make the playoffs has lead to a number of eighth seeds with losing records that have made it deep into the playoffs (including appearing in two of the last three MLS Cups). In this post, I will explain how the disparity in the number of games played throughout the regular season contributes to the predictable outcome of a first round playoff match, and why it seems to uniquely penalize the higher seeded teams.

Factors In Predicting MLS First Round Playoff Outcomes

Climbing the Ladder had a great post right after the conclusion of the first round of the 2010 playoffs were different factors that had contributed to first round playoff success since 2003 were compared to each other. A snapshot of the table that was developed is shown below.

What was interesting is that fewer games, out of all the factors studied, stands out as the most significant determinant of first round playoff success. This seems a bit unexpected to those versed in how the US game operates, and I will explain why in a bit. The next most significant factor – regular season goal difference – does make a good bit more sense. The less significant coach field is much like the fewer games category - the results are a bit opposed to what our first guess might suggest. As the comments in the original post evolved, it became clear that everyone wanted to focus on the “fewer games played” category. Here’s why.

While MLS plays a balanced schedule in their regular season, their teams have three other types of matches they can play.

• Friendlies: The number of friendlies a team plays is under the direct control of the team management, and varies from team-to-team. Teams may use them as pre-season conditioning or as mid-season revenue generators. I’ve seen both approaches from my hometown team, the Seattle Sounders. Regardless, they demand match preparation and player conditioning, even though the starting XI may get changed from the side that participates in a typical MLS match. While teams somewhat enjoy them, a growing number of fans aren’t. Seattle’s supporters have been so vocal as to create a Twitter hashtag to emphasize their displeasure, which was subsequently picked up by Real Salt Lake. Perhaps the supporters are on to something – if you’re going to lower the chances of the team making it deep into the MLS Cup Playoffs, don’t do it with a meaningless friendly.
• CONCACAF Champions League (CCL): Like every other continental governing body, CONCACAF has a tournament where the best teams from each league play each other to determine which club is the best from that governing body. Winners of the CCL move on to the Club World Cup. While the format for the CCL has changed over the years, it has followed a 4 team-per-group, 6 game balanced-schedule-within-group format since 2008. While the standards for MLS CCL qualification has also changed over time, currently the MLS Cup champion and Supporter’s Shield winner are guaranteed a spot in group play, while the MLS Cup runner up and US Open Cup champion must play a two-legged play-in via the preliminary round. This means MLS CCL participants can play an additional two to eight matches during their MLS regular season, depending on qualification path. As the CCL follows the FIFA calendar, the six games a team may play in the Championship Round of the tournament occur at the beginning of the next MLS season. Thus, if a team qualifies for the Championship Round in one CCL tournament and qualifies for the Preliminary/Group Play stage in another, they may play up to fourteen additional matches in a season.
• US Open Cup: The longest running soccer cup competition in the United States has been contested since 1914. Thirty-two teams from the lower divisions of the American soccer pyramid begin the tournament in June, with eight MLS teams being added to the third round to compete against the eight remaining teams from the lower levels of the pyramid. MLS teams must play four matches in the tournament to win the US Open Cup. MLS assigns its berths in the tournament by giving its top six teams (by points) automatic berths and the remaining teams compete for a berth via a playoff format. Thus, MLS teams can play between four and eight matches in qualification and participation in the US Open Cup.

Adding up all these competitions, an MLS team can play between 1 and 22 additional matches in a season (not including friendlies). The reality is that with so many MLS teams competing in each competition and each team making it to various stages of each tournament, the disparity in games played between each team is much lower. The highest game difference was the 2008 first round match-up between Columbus and New York where New York had played fourteen fewer games. The average difference in games played between first round opponents since 2003 is 3.28 (median of 2.0).

All of this makes for a pretty compelling case that game differential may play a significant roll in the first round. This seems intuitive – teams that make it deeper into competitions outside of MLS will have played more matches, and fatigue at the end of the long season may take a bigger toll on those teams. Combine fatigue with the wild nature of a playoff, and you have a recipe for the team who has played fewer games to win. This seems a bit unfair, as the only way these teams have played more games is that they finished well the season prior (qualifying for the CCL) and/or have played well in the US Open Cup (in trying to qualify for next season’s CCL). The US soccer landscape already has two knockout style championships, so why does MLS feel the need to add the unpredictability of a third?

Testing the Data Statistically: How Playing Fewer Games Leads To Upsets

All of those observations are just idle speculation if they can’t be backed up by statistics showing the difference in games played actually upending expected results. Interestingly, the statistics present a mixed answer that suggests MLS may just be proceeding down the path of unintended upsets in how their playoff structure interacts with the teams’ other competitions.

The first comparison that was studied was Climbing the Ladder's observation regarding the overall impact o fewer games. In this case, it must be determined if the sample size is large enough (enough playoff games played since 2003) to determine if the observed effect is statistically significant. This entails comparing the expected percentage for overall first round playoff matches (50% chance of winning the series) to the observed percentage for teams that win and have played fewer games throughout the season (67.2%). As my comment in the original Climbing the Ladder post suggests:

Sadly, even the "fewer games" category is not statistically significant when compared to the 50/50 chance most teams have of advancing. Using this site and entering the following values:

p0 = 0.5

p1 = 0.672

1 sided Test

Default alpha and power levels

Yields a sample size of 51. This means this proportion of win percentage would have to be viewed over 51 matches to determine that fewer games was a good predictor vs. a coin flip.

This means that the 32 opening round series played in the last eight seasons and the resultant 67.2% win percentage for teams that played fewer games during the regular season are not significant enough to declare an advantage for such teams. Statistically, such a percentage is technically no better than flipping a coin to determine who moves on to the next round.

However, interesting conclusions can be drawn when the focus turns from the overall "fewer/greater number of games" metric and instead look at the impact of the difference in the number of games played. One way to measure this is to look at each series where there was a difference in the total number of games played by each team, and note if the team with fewer games won the series or not. Teams that had fewer games and won are binned as a "correct prediction", while teams that lost while playing fewer games during the regular season are binned as "incorrect predictions". If each bin of predictions is then sorted by ascending goal difference and a (1-CDF) plot is generated (a concept introduced here), the effect of increasing goal difference in predicting the outcome of the first round playoff series is understood. A graph representing this relationship is shown below (click to enlarge).

What becomes immediately clear is that by the time the game differential gets to four, the team with fewer games has nearly a six in seven chance of winning the series. No team who has played six or more games during the regular season has won a first round playoff series. Clearly, three matches or fewer appear to be a statistical toss-up, but a match difference of four or greater greatly favors the team that played those precious fewer matches.

Recall that the MLS Suppoters' Shield and MLS Cup winners receive automatic berths into the CCL. The odds seen in the (1-CDF) plot also play out when we look at these two types of teams that also tend to play a much greater number of games.

The table below summarizes the performance of each Supporter's Shield Winner once they reached the playoffs. Five of the teams played no more or less than two games when compared to their opponents, and this resulted in mixed results for the team that played fewer games. The three series where a team had played four or more games less than their opponent saw that team win the series. The latest occurrence of this is especially illustrative. The 2010 playoffs saw the Los Angeles Galaxy defeat the Seattle Sounders after the Sounders had played two more games than LA in the US Open Cup (in fact, Seattle eliminated LA in that competition) and Seattle played six more games in a poor CCL group play campaign (due to LA failing to win their preliminary round series).

What's even more interesting is studying the performance of MLS Cup winners in their first round playoff matches when defending their title (see table below). Save for LA in 2005, each MLS Cup winner since 2002 has qualified for the playoffs the season after they won the championship. Nearly every one of them lost in the first round, with the Houston Dynamo being the only exception when they successfully defended their championship in 2007. Ironically, Houston would follow up that 2007 campaign with a 2008 campaign that bridged the prior and current CCL format, which meant they set a record for first round game differential (14) that will likely never be equaled. They subsequently lost to New York in the 2008 playoffs.

Ultimately the table confirms what the graph showed. The two first round series that had a game differential of less than 4 were a toss up - one win and one loss. The remaining series all involved game differentials of four or more, and all five ended in favor of the team with fewer games played. With the latest format for the CCL, where MLS Cup winners automatically qualify for six matches of group play rather than a home-away elimination format as in years past, we've likely seen the end of any dynastic teams like those in the first decade of MLS.

The predictability of outcome of first round series with a game differential of four or more does not stop there. Of greater concern is the alignment of such predictability with the likelihood of an upset. Each of the thirty-two first round matches was categorized as an upset if the lower seeded team won (as indicated by conference seeds one through four, regardless of whether or not a team was moved from one conference to the other under MLS's odd playoff seeding rules). The table below shows the results of each first round playoff series, categorized by whether or not an upset occurred and binned according to the difference in the number of games played.

Clearly, as the game differential increases the likelihood of an upset increases. If a comparison is made between the ratio of upsets in the series with a game differential of four or greater versus the rest of the series, one finds that there is insufficient sample size to determine that the difference is statistically significant. A further breakdown of the matches with a game differential of four or more also yields the table below.

What is interesting is that matches with a game differential of four or more have picked up significantly since the change in the CCL group play format in 2008. In fact, using this calculator to make a comparison of the proportions of the table above shows that a statistically significant difference in the two eras exists if one more series were played in each era and the proportions held. The proportion of matches with a game differential of four or more has certainly increased since 2008.

The proportion of series that ends in an upset has also gone up with era, but not by a statistically significant amount.

Conclusions

In general, it appears as if there is a greater number of first round series with a game differential of four or more games since 2008. Recognizing the challenge such a game differential presents to a team is key in setting realistic expectations for a season. Several key conclusions can be drawn.

• If a team wins the Supporter’s Shield and/or the MLS Cup one season, they are unlikely to make it out of the first round of the next year’s playoffs. Sorry LA Galaxy and Colorado Rapids supporters.
• Fans have to be realistic when it comes to setting expectations for success in MLS and other competitions. My fellow Sounders Supporters have the desire to pull off the domestic double and possibly the CCL treble. This is completely unrealistic in MLS – supporters need to pick which competition they want their team to win and not expect the impossible.
• Sounders supporters, who desire a US Open Cup threepeat, should think twice if they want an MLS Cup. Such a win would put the team back into the CCL and likely cause them cross the crucial four match threshold.

The causes for this correlation are not clear from the statistics, although I do have several theories. Extra-league competitions are the norm the world over. What is not the norm is MLS’s salary cap and the limitations it places on the quality of the players in the league. While overall player quality is impacted, I suspect it is most pronounced in the supporting members of the starting XI and especially in the substitutes that might be expected to see more action in US Open Cup and CCL competitions. I suspect that this limits the quality of the players that teams can rotate through the various competitions they compete within, thus creating a greater reliance on a core XI and more fatigue in the players normally used for league play.

Also likely is a good bit of correlation with overall parity for which MLS is notorious. As the Sounder at Heart blog suggests, they only measure how much parity there is and not whether or not parity is good. Interestingly enough, the English Premier League scores at the opposite end of the parity index. This aligns with less numerical expectations, and illustrates the conundrum I have. I want a meritocratic league where teams are allowed to reward players with the best contracts possible, and I want players to be rewarded financially for their skills. In the same vein, I don’t want to turn the league into such a financial boon that Russian oligarchs or oil sheiks decide to purchase championships for their their clubs with funds from outside their soccer operations and thus drown out other clubs from competing for available talent. In the case of MLS, the seemingly random nature of the league and the increasingly predictable success of the lower seeds in the playoffs suggests that the system needs to be corrected.

The exact nature of that correction can take many forms, but I will attempt to make some suggestions in my next post in this series.

An Indictment of the MLS Playoff Structure, Part 1: Letting Too Many Teams in to the Party

Note: This the first of a several part series that I will be writing about MLS's playoff structure and how it penalizes successful teams.

Pay As You Play begins its concluding section with the following observation.

"Contrary to popular wisdom, things don't even out exactly - after all, no higher power exists to bring immaculate balance (and if there are football gods, one assumes that they exist purely to mock us, not provide parity on refereeing decisions) - but a whole campaign is a better barometer for the quality of a football team or a manager than one or two games."

Sadly, that's not how we determine our league champion in the United States' top flight of soccer. Our little revolution in 1776 has led to an odd cultural habit in our country - we refuse to ever adopt something as-is, often insisting on some uniquely-American twist to a perfectly fine product. This even applies to our soccer league. Sure, we've gotten rid of some of the worst examples of such attempts at creating "American soccer" (backwards counting clocks, insisting on eliminating the tie, etc), but the abomination that is a playoff-based championship persists and it penalizes successful clubs. Let me explain...

Some Background on US Professional Sports and Playoffs

A playoff-based championship format makes sense under two conditions:

1. When a league is too large to play a balanced schedule, OR
2. When the time constraints of the competition do not allow for a balanced schedule.

Scenario (1) applies to much of the professional US sports landscape. We're a nation of 300+ million people, we rank third or fourth in land mass (depending on the calculation method used), and most of our leagues are well over 20 teams. The National Football League (NFL) has 32 teams, and the National Basketball Association (NBA), Major League Baseball (MLB), and the National Hockey League (NHL) each have 30 teams. Travel from coast-to-coast can take 4-5 hours on a direct flight, with a three hour time change also involved. Beyond the geography, the each league would require between 58 and 62 games in a balanced schedule.

Given these difficulties, each league has split themselves along conferences and subdivided divisions, as well as adopted a playoff system. The format ensures that each team plays a balanced schedule within their division, and then a rotating batch of teams within their conference and across conferences. This overall unbalanced schedule then demands a playoff to decide the champion, with seeding determined by each team's relative performance during the season. Each of the leagues break a differing percentage of their teams to the playoff rounds:

• NFL: 37.5% (8/32 12/32)
• NBA: 53.3% (16/30)
• MLB: 26.7% (8/30)
• NHL: 53.3% (16/30)

Each round of the leagues' playoffs consist of a best-of-five or a best-of-seven series of games (the NFL is the one league that doesn't, opting for a single elimination playoff). Using the a five or seven game series minimizes the chances of a team pulling off a string of a few upsets to win a championship, and instead forces teams to play a minimum of 12-16 games to win a title.

The MLS Playoff Structure

The top flight of US soccer, Major League Soccer (MLS), also uses a playoff system to determine its champion. For a number of years MLS has been admitting eight teams each season into their championship playoff system (in 2011 they will admit 10 teams), even when they reached their modern era low of ten teams after contraction in the early 2000's. I have focused on the 2005 season onward, as the related financial data I use in other analyses stops being reliable any further back in the league's history. From 2005 to 2011, the league has increased from 12 to 18 teams. This has generated the following percentage of teams making it into the MLS playoffs.

• 2005: 66% (8/12)
• 2006: 66% (8/12)
• 2007: 61.5% (8/13)
• 2008: 57% (8/14)
• 2009: 53.3% (8/15)
• 2010: 50% (8/16)
• 2011: 55.6% (10/18)

The tournament is a bit of a hodge-podge of elimination formats. The initial round is a two match home-and-away format with the team with the highest aggregate number of goals moving on to conference finals (penalty kicks are used as tie breakers). The final two rounds - the conference finals and the MLS Cup - are both decided by single elimination matches. Since 2007, the league has used the following format to determine seeds:

1. The top two finishers in each conference (MLS has two - East and West) are automatically seeded as #1 and #2 in each conference bracket (overall seeds 1 through 4)
2. The next four seeds are populated with the teams with the next four highest point totals, regardless of which conference they are from. These teams can be referred to as wild cards. If a conference's four spots are already filled by their top two finishers and two of the wild cards from their conference, any remaining wild cards from that conference will be placed into the opposite conference's bracket.

This structure provides for some interesting pairings. First, it doesn't seem to reward outstanding play of top finishers in a conference. In all other leagues, the reward of being seeded #1 in the playoffs is getting to play the lowest seed. That is not the case in MLS where situations like the 2010 season arise - the two Eastern Conference qualifiers got to face Colorado (#7) and San Jose (#8). Meanwhile, Los Angeles and Real Salt Lake, who fought to the last week of the season for the top-of-the-table position, were rewarded for all their hard work in facing the hottest team in the league (Seattle at #6) and a team whose home pitch was a fortress all season long (Dallas at #5).

Second, it has resulted in the last three MLS Cups having teams from the same conference facing each other for the league championship. One might question the fairness of a playoff system where a team who has underperformed all season long gets to cross over to the theoretically weaker conference (assuming that the conference with fewer teams is weaker) for their playoff run. No one would make the serious argument that New York (2008), Real Salt Lake (2009), or Colorado (2010) were one of the top four (let alone two) teams in MLS each of the seasons they reached the final. Yet those teams got a shot at winning the league championship after underachieving all season, with Real Salt Lake and the Colorado Rapids each walking away with the title.

If the standard of admittance to the MLS Cup playoffs is so low, what quality of play is required to reach that low standard?

Points and Goal Differential Required to Qualify for the Playoffs

What exactly does it take to qualify for the MLS Cup playoffs? Luckily, data compiled from the 2005 season onward can give us insight into the typical points and goal differential required to finish eighth or higher in the overall league table.

Given the increasing number of teams since 2005 and the varying level of matches played in a season, each of the variables has been normalized. Finish position was translated via the now-familiar Soccernomics natural logarithm transform, which also provided a good way to assure the data set was normally distributed. Each team's points were translated into a percentage of the available points earned in a season. The plots below show the relationship between each metric - finish position versus % of points earned, finish position versus goal differential, and % of points earned versus goal differential. Click on each image below to enlarge them.



















Now that the relationship between points, goal differential, and finish position is understood, the performance required for an 8th place finish and qualification for the MLS Cup playoffs can also be understood. Applying the equations in the regression plots to each season produces the table below.







What one immediately sees is the low quality of teams that will typically qualify for the MLS Cup playoffs. A losing record (< 50% of available points earned) can be expected, and so can a negative goal differential. The requirements of a typical 8th seed were getting more difficult from 2005 through 2010 as the number of playoff spots remained the same while the number of the teams in the league increased, so that by 2010 the typical team would have earned 46% of the available points and had a positive goal differential. Fans were disappointed when the MLS commissioner announced that the playoff field would be expanded to 10 teams in 2011, lowering qualification standards back to 2008 levels.

When examining the actual points accrued and goal differentials of the 8th seeded teams from 2005 to 2010, the picture looks only a little bit better. The table below (click to enlarge) summarizes their results versus the regression equations, with the yellow columns representing the data for each of the teams.

As would be expected, three of the teams earned a percentage of points above the predicted value and three below. San Jose's 46 point haul over thirty games in 2010 was the best performance of an eighth seed over the six years. This should come as no surprise as it was also the most lopsided table over the period with six Western Conference teams qualifying for the playoffs, a result of the Western Conference's domination throughout the season.

The results for goal differential look like a great overachievement, but they mask the teams around the 8th seeds that had substantially worse results. In 2006, Colorado was tied on points with New York and had a -13 goal differential. In 2007, Chicago tied Kansas City on points and had a -5 goal differential, while FC Dallas was ahead of Kansas City with 44 points yet had a -7 goal differential. Kansas City, New England, and Chivas all qualified in 2008 with negative goal differentials. New England qualified again in 2009 with a negative goal differential, this time at -4.

What's even more disturbing is the final column of the table, which indicates the final round in which each team participated. Save for New York in 2006, each of the 8th seeds has made it out of the first round. Three of them have gone on to the MLS Cup, with two of them winning the Cup (LA in 2005 and Real Salt Lake in 2009). It seems that most teams that have qualified eighth have been able to pull off at least two games above their seasonal average poor performance and move on to the next round of the playoffs.

Preliminary Conclusions

It seems as if MLS is not only pretty generous with who they let in to the postseason, but their playoff format almost seems to favor teams that have underperformed. Without a multi-game series that gives the higher seeded team some sort of advantage (home field, first game being away and counting away goals more than home goals, etc.) there seems to be no benefit in the MLS playoffs to performing better during the regular season. We Americans love our postseason tournaments - arguably the biggest sporting event each year is known as "March Madness" for all the upsets it produces. This style of single elimination playoff championship is only found in one of the four major US professional sports leagues, and it is not the way MLS should be determining it's champions after nearly seven months of a balanced regular season schedule.

All discussion to in this post has been based upon point data - single observations or groups of observations but no study of the statistical distributions. What really drives success in the MLS playoffs, especially the critical first round where upsets seem to abound? Luckily, someone else has compiled good summary data of what can be used to predict first round success. The topic of what seems to predict first round success will be the studied in detail in Part 2 of this series.