BoA and LT will cover game-day stuff better than I; this simply seemed like a reasonable excuse to post the information I've been promising for a couple of days on preseason success and why it doesn't mean shit.

Last week, Ender jokingly suggested to me that maybe the Oilers' losing was a good thing, because preseason success seemed to be a harbinger of regular season failure, and vice versa. It was a fun thought, and certainly, one look at the 2006 preseason should tell you where that idea comes from. On the other hand, it seemed a bit illogical: surely there shouldn't be any relation at all between pre- and regular season results, should there? I mean, the good teams will probably still be good, and the bad teams will probably still be bad, but there's a lot of room for slosh in a sample of only 5-8 games: the Oilers have started many a season firing on all cylinders in October and November before coming back to Earth in January, while the Flames have started all three post-lockout years with their skates tied together, and made the playoffs every time, winning the division in 2005-06.

It wasn't an idea that I gave much further thought to until Tuesday night, when the Oilers got clubbed 4-0 by the Flames, and did so while looking pretty much like total shit the whole way through. So I decided to see if there was anything to this idea, and grabbed the last three years' preseason data, fired that into Excel with the regular season standings, and checked to see what came out of the wash.

Part 1: Visual Inspection

Taking a page of sorts from LittleFury, I made the top 10 teams in the preseason, by points percentage, green, the next ten yellow, and the final ten red with white text. The teams are sorted according to the NHL's own standings. The results are below.

While there is somewhat of a trend for good teams to clump a bit at the top, and bad teams to clump a bit at the bottom, there's all kinds of spread here. There's top five preseason teams in the bottom five of the regular season standings, and vice versa. You could maybe say that 2005-06 "looks" less messed-up than the rest, but I don't think you can say that with any conviction, and I don't think I can even honestly say that I see it, since I was spoiled by doing the second test before I got around to making those highlights.

Part 2: Linear Regression

After that, I did a quick linear regression to find out if there was any linearity to the data, and if that linear trend pointed in a given direction. The figure below shows plots for the three seasons, and trendlines for the same, as well as a line for the pooled data of all three years.

That would be a big, fat nothing. There's a very small but significant (p = 0.016) correlation between preseason and regular season results in 2005-06 (R² = 0.19), which as I noted before, you might almost be able to see in the raw data if you squint really hard at the table, but the other two years are below 0.04, with no significance whatsoever (p >> 0.05 for each of them), with the three-year pool clocking in with an R² value of 0.07 (I ran out of time to run signficance on this, though I'd lay money down that it's not a significant trend). Between that number and just looking at the scattershot nature of the data, it seems fairly obvious that there's no real connection between preseason and regular season success.

I was going to do further statistical testing, but (a) it seemed pointless by this juncture, and (b) I couldn't find anything that worked quite the way I needed it to other than -- you guessed it -- linear regression. So there we stand.

Limitations

It's somewhat ironic that the greatest limitation of the study -- the small sample size of the preseason -- is also one of the reasons why there is a difference. I mean, five to eight games can easily deceive you in a way 82 can't. While there's all sorts of problems with even a full-season sample, it's not nearly as problematic as a stretch that comprises 6-10% of a season. I said what I needed to say on this at the top of the post, so we'll move on.

A second problem here is that I only have three years' worth of data here. Obviously, pre-lockout data would complicate matters, due to the differing rulesets, but if I had to guess, I'd say that it probably wouldn't make that much of a difference -- the mean points percentage would drop a smidge, but otherwise the trend, or lack thereof, should probably be about the same -- but even aside from that consideration, it's not easy to find pre-season results, especially the further back you go. Yahoo! has 2003-04, but I would rather have all the data from the 30-team era rather than just the one year. The net effect of all this is that I can't definitively say that the significant trend from 2005 is the outlier and the clouds of nothing from 2006 and 2007 are the reality, because it's two years of one and one of the other, and because I haven't run anything prior to 2005, but it passes the smell test, so I'm gonna go with it, at least for now. I'm not sure why 2005 was so special: it was power-play city that year, so that could be a reason, though I'm not sure why special-teams play would be any more significant, or any more likely to carry forward, than final results. It's something that'll require further study, though again, I'm not sure how to get a hold of that data.

The final and most galling limitation is the Bettman point. Yes, the sometimes-two-sometimes-three-point system rears its ugly head again, because it kept me from running a chi-square test on the data, which I kind of wanted to do as a third check. Since the chi-square test checks the distribution of the experimental data against a model prediction (in this case, the preseason being the "model" and the regular season beng the "experiment," with the null hypothesis obviously being that the preseason predicts the regular season), I needed the two columns to have the same sum, which was not, of course, the case. I did a bit of fiddling with the numbers, including "percentage of points awarded," but I couldn't get anything that really worked well, so I eventually abandoned the pursuit, which is too bad, because it really would've completed the set.

Interpretations

Now that we've got all that out of the way, what can we actually say from this? Put simply, the preseason has no bearing on regular season results. You could argue that, depending on how a team played its last couple of preseason games, it might be an indicator of how the early season goes, but even that seems like a pretty sketchy proposition, when you consider what all goes into a preseason record.

For starters, there's the rosters. Specifically, the rosters that frequently have little more than the minimum required number of NHL veterans (eight) in the first half, when coaches are usually more interested in development and scouting than winning. You cannot reasonably judge a team of 23 NHLers based on the work of up to 13 guys who might never see a minute of meaningful NHL time, and if they do, it likely won't be in the tenure of the current coaching and management crew, given how long those guys tend to last in most NHL cities. This is also why I really couldn't be bothered when the Oilers rookies went 0-2 in their own tournament, yet beat both the U of A Golden Bears and the ACAC All-Stars. It doesn't involve players who will actually be deciding the games that matter, so why sweat it? Even later in the year, there's enough non-NHL players involved that it's hard to take the results as being indicative of anything; kind of like the last 20 games of the 2006-07 season.

Related to this is the schedule. The preseason frequently starts with crazy stretches like four games in five nights or five in six, to say nothing of the stuff they've been doing in Europe the last couple of years. Teams will often play split squads during those stretches, with minimal overlap between sub-teams, and certainly none over more than two consecutive nights. One team might have more vets than the other, and be an "A" team of sorts, while the other has more kids and scrubs, a "B" team. A-Teams, as a general rule, will clobber B-Teams, though on any given night, anything can happen. The results of both the A- and B-Team will be combined, since they're all wearing the same shirts, so you have no cohesive whole to make your comparison with, and frequently less-than-fair competition to generate the results against in the first place. Granted, you don't exactly have a cohesive whole during the regular season, either, with trades, signings, callups, reassignments, and injuries all tweaking the roster throughout the years, but it's at least mostly cohesive, unlike the preseason, when most of the roster can change from one night to the next.

Next, there's coaching. As noted above, training camp is a time for development and scouting. Coaches will do crazy shit like playing 19-year-old sophomore scorers on the penalty-kill, or break up established lines and add kids and minor-leaguers to the mix, for the sake of seeing what these guys have got, and if anything good comes out of the wash. I guess it's the Yukon Cornelius approach to talent evaluation: you never know when you might accidentally strike gold while randomly throwing your pickaxe in the air (see: Thoreson, Patrick; Gagner, Sam). Anyway, when lines are out of whack, kids are playing tough minutes, and guys who don't normally kill penalties are out there doing just that, you get results that are in no way indicative of how the team is actually going to play when the games count. How Craig MacTavish coaches against the Florida Panthers in September is nothing like how Craig MacTavish coaches against the Minnesota Wild in March.

Finally, we have the players themselves. Guys get rusty over the summer, and need time to get the system back in their heads and the chemistry back with their linemates. They're probably going to be going for a slow burn, to ease their way back into things, so they don't hurt themselves by doing too much, too soon. Coaches don't want key players to get hurt, either, something the Oilers have learned from past experience, and something the Penguins are learning now with Gonchar gone for most of the year. Sure, players are going to want to be at 100% by the time the season starts, but if they play their first preseason game at 60% or 70%...does anyone really care? Obviously, if you lay an egg like the Oilers did on Tuesday, no one's going to be too thrilled with it, nor should they be, but it's hardly a sign of the Apocalypse on Ice. Also, much like during the regular season, you have to trust that after a horrible effort, the professionals will be able to get their shit together and do it right next time.

Conclusion

So what does all this mean for the Oilers, in the end? Absolutely nothing. Maybe they'll be good. Maybe they'll struggle out of the gate, then turn it around. Maybe they'll be Godawful. No one can know for sure until the games are played, though we all have our ways of making educated guesses (and speaking of, I guess I have more work to do this weekend). What this post should prove, however, is that preseason results shouldn't be one of those ways, or at the very least should be used with extreme caution, since you could probably throw darts at a chart of the 30 teams and be just as likely to pick out the final results from that as you would from the pre-season table alone.

Comment on this post...