When Small Samples are the Best Samples
We’re a quarter of the way through the MLB season, and while some stats are stabilizing, many are still highly influenced by noise and variance. We’re also six or so weeks into the minor league season and many sites, including this one, have updated their prospect rankings. There’s also been some discussion both around the interwebs and in the Dynasty Guru Slack about the value of this data and the usefulness of updating prospect rankings more than twice per year. In this piece, I’ll be trying to offer some context on how, when, and why to change your opinion on a player.
All you hear from fantasy experts in April is “SMALL SAMPLE SIIIIEEEEZE!!!” screamed at the top of their lungs. Once you get to May though, you start hearing talk of stats “stabilizing” and becoming meaningful. I’m not going to go into detail about when each and every stat stabilizes; go read this great FanGraphs primer on that here if you want that. The short-and-general version is that some stats reach meaningful samples sooner than others. What I want to talk about here is the difference between evaluating established MLB players and prospects (statistically speaking).
When looking at a major leaguer, we usually have a fairly hefty career of stats to use as a baseline. If they start doing something a lot different from that, we need to find strong evidence of a material change before we can believe that the small sample is anything but an aberration of randomness (That’s also the name of my procedurally-generated prog-rock band). That’s because six weeks of games in a five-year career is a drop in the bucket.
For example, Paul Goldschmidt is having a bit of an off year thus far, hitting around .200 rather than the .300 he typically puts up. This stretch doesn’t change what we think Goldschmidt is, and if we can’t find a reason for his struggles, we’re going to chalk it up to bad luck and say he’ll be fine. Goldy’s early career is also a great example of us moving too slowly on a guy who demolished minor league pitching at every level without missing a beat.
We don’t have that kind of solid baseline when looking at a prospect. Even if we have 5+ years worth of games, they tend to look something like Lewis Brinson’s minor league career:
That’s just a mess to look at. Eleven stops in six seasons spread out in almost every way possible. Multiple levels, split seasons, he only missed different Double-A leagues in the same year because the Brewers sent him to Triple-A after his 2016 trade. You can’t just add them all up and know who that player is; there are too many contextual factors to account for first.
Most top-100 prospects don’t even have that much data though. The majority have between zero and three years of pro experience, so the aforementioned six weeks of stats for them is less a drop in a bucket and more a drop in a shot glass or a tall boy. That’s not a great metaphor, but the point is that a small sample means more when you have less data to start with. When a player has very little experience, we tend to lean on pre-draft scouting reports and tool grades. Those are useful as a surrogate baseline expectation for that player, but even those are based on limited looks against wildly different levels of competition.
Another big difference between an MLB career and a minor league career is the context of those stats. Let’s continue our hypotheticals from earlier: a five-year major league career is going to come against more or less the same level of competition, with fairly mild age effects. Prospects, on the other hand, go through major changes in their bodies as they grow from teenagers into large adults. At the same time, they’re moving up the minor league ladder, and age relative to level is a significant factor at every stop. Lewis Brinson in rookie ball at age 18 isn’t really comparable to 20-year-old Brinson in High-A or 22-year-old Brinson in Triple-A. That’s not even taking into account the mechanical tinkering that players and coaches are doing along the way, or the simple developmental benefits of getting more reps.
We do tend to afford prospects an “adjustment period” every time they jump a level, so we don’t want to be too hard on them in their first month or two in a tougher league. Aaron Judge did this consistently, struggling at every level at first, then adjusting and mashing until he got promoted. The same thing happened in his first taste of the Majors in 2016, followed by an adjustment and him mashing in 2017.
On the flip side, we should be impressed when they move up without missing a beat. Yes, I’m looking at you, Vladimir Guerrero Junior and Juan Soto (Austin Riley, Brandon Marsh and Shane Bieber are also good examples of this). What all this adds up to is that we should take recent data more seriously with prospects because the context is more relevant, and we can generally assume that they’ve gone through significant physical and mental/skill changes along the way. Human beings are not static, they grow and change, and that’s especially true with prospects.
Small Samples and Prospect Lists
There are many outlets (this one included) that have updated their pre-season prospect lists, and there are several that haven’t. From a “Real Baseball” perspective, I fall on the more conservative off-season/post-draft update cycle side of things. There’s more risk of looking like a fool for hammering a guy for a slow spring. There’s also less to gain from being in on a guy a month or two ahead of the pack. I certainly get why major sites do it (page views) but I also get why you wouldn’t.
The “Fantasy Baseball” perspective s a different animal though. In dynasty leagues, prospect lists are like stock tickers. We’re a bit less concerned with the ultimate career value of a player and more focused on their market value. What do we have to give up to acquire Brendan Rodgers? What can we get for Eloy Jimenez if our MLB team is hurting? In that sense, monthly lists are much more valuable to dynasty owners than to general baseball fans. I also don’t think either approach does your readers a disservice, so long as you’re “showing your work” with more detailed player reports and looks along the way. Not everyone has time to read those, but you don’t need a list update to be “in” on Juan Soto as a top-5 prospect in May (though if you haven’t written him up yet, you may be too late).
Scouting and Market Value
I am not a scout. My eyesight is terrible and while I have been watching baseball for almost 30 years, I haven’t been watching it the way a professional scout does. All prospect writers, even if they do live scouting themselves, are going to check their impressions of a player against other scouts whose opinions they trust. Those scouts’ opinions are more data points that help paint a more complete picture than a single scout or set of stats could. We read other experts, who read other experts, or have scouting contacts. We try not to just regurgitate (or plagiarize) another writer’s work, but we can’t help but be influenced by their opinions. When we are talking fantasy ranks, those opinions influence public perception of prospects, and we have to take them into account if we want to provide an accurate market value for a player. When a prospect gets a write up on a major site, there’s a good chance that their owner in your league is going to read it.
What I try to do – and what the industry as a whole could probably do better about – is to directly reference and credit the people whose work contributes to ours. It takes a little extra time, and sometimes it’s difficult when it feels like everyone is writing about Vlad or Soto, but it is our responsibility to give credit where it is due.
All Data is Data
The takeaway here is that all data contributes to the bigger picture of a player. The larger the existing sample, the more new data it takes to change that picture. With prospects, we tend to have less data to start with, and the value of that data degrades more quickly thanks to age/level factors. We should give greater weight to new data against tougher competition, and not be so afraid of recency bias as we are with major league players. We still shouldn’t overreact to six weeks, but failure to incorporate that new data at all is just as big of a mistake.