Update (4/1/19): Name has been officially changed to CRA
Welcome to the unveiling of CRA, or Classified Run Average, a new ERA estimator that includes various xStats for a better, more realistic idea of how a pitcher’s average, batted ball results impacted his overall stat line.
Why a New ERA Estimator?
This all started with a little conversation from a Twitter account that I enjoy interacting with, @ThatsaBadTake. He’s a Yankees’ fan, unfortunately, but he’s a smart guy and knows his stuff, so the interactions are always enjoyable, if not thought-provoking. We were discussing James Paxton and, more specifically, his 2018 season.
Here are Paxton’s 2018 numbers: 3.76 ERA, 3.24 FIP, 3.02 xFIP, 2.96 SIERA
The conversation was, essentially, was Paxton closer to a 3.75 ERA pitcher or was he closer to a 3.00 pitcher that his estimators had him as in 2018? The estimators assumed that his results on balls in play and HR/FB were on the higher side and should regress to league average. But should we just assume that? That question was at the forefront of my inquisition.
Based on Baseball Savant’s xStats, which incorporates exit velocity and launch angle to produce an expected batting average (xBA) and expected SLG (xSLG) based on similarly hit balls, perhaps we shouldn’t, in Paxton’s case. Paxton’s exit velocities last season were quite high, and it resulted in a .337 xBA on contact (xBAcon) and a .581 xSLG on contact (xSLGcon), both numbers not very good compared to his peers.
What CRA attempts to do is remove some of those aforementioned assumptions from the equation. With Statcast, we now can safely, roughly quantify how hard a pitcher was hit, and intuitively, a pitcher that was hit much harder than another pitcher is going to have worse results on hit balls. So, with Paxton, the assumption that his results on batted balls should have been closer to league average is possibly incorrect. CRA attempts to capture that.
When I first started with this mini project, I knew I wanted to capture 3 main things; strikeouts, walks, and the quality of all batted balls independent of outcome. That will, for most pitchers, capture 98% of PA with the exceptions being HBP and silly things like catcher interference. So I started with 4 main variables: K%, BB%, xBA, xSLG.
Very quickly, as I began to share some initial results, Jeremy Frank (another good Twitter follow, @MLBRandomStats) pointed out that perhaps I should be using only xBA and xSLG on only batted balls, otherwise strikeouts are incorporated in 2 separate variables (once in K% and once in xBA). I’m sure the model already accounted for this, and I didn’t go back and take a look at exactly how much this changed the model, if at all, but I agreed it made more sense intuitively, so I changed the final two variables: K%, BB%, xBAcon, xSLGcon.
Long story short, Games, Games Started, and xwOBA were also added to the model, as they made it ever so slightly better, but the 4 main variables above were the meat and potatoes, and carry something like 97% of the weight.
Based on the new model, Paxton’s CRA - which, as stated, includes his K%, BB%, and expected results based on the quality of batted balls allowed, was 3.54 for 2018. In 2016 and 2017, his CRA was 3.32 and 2.73.
Now, the CRA model is not a prediction tool. It was not designed that way. It is an estimator for what should have happened. CRA simply gives a number that it thinks the pitcher’s ERA should have been close to based on performance without attempting to predict what should or will regress and what should not.
That said, early predictability testing on the metric has given surprisingly good results. The root mean squared error (RMSE) of CRA to the next year’s CRA is 1.23, meaning +/- 1.23 is the average error between the predicted value and the very next season’s residual (ERA). If it seems high, it’s because it is; predicting ERA is very, very hard. However, with context, it’s a good number:
The RMSE year to year between ERA and ERA is 1.47.
The RMSE year to year between FIP and ERA is 1.27.
SIERA is best at predictability, as the RMSE between SIERA and next year’s ERA is 1.16.
For a first iteration, getting a smaller average error than FIP, one of the game’s most well known estimators, is a definite success in my opinion.
By ERA and CRA standards, James Paxton had a down year. The 3.54 CRA is not intending to say that James Paxton will continue to be a 3.54 pitcher going forward, just that he likely was a roughly 3.54 pitcher in 2018. Regressions in one or all of these variables, for better or worse, should be expected, and his CRA should follow. In fact, based on prior CRAs from 2016 and 2017, we might even go as far as saying Paxton should be expected to be back into that 3.30 range or better in 2019. Should his batted ball data stay high, his CRA and probably ERA will as well.
Hopefully, future iterations of the model will include sample size and regression analysis for each archetype of pitcher (similar to how SIERA does) to improve the model. For now, Paxton’s 2018 number gives me hope that iteration 1.0 is working as intended.
Where CRA Works Well
Simply put, CRA works well for pitchers that have a very high or very low xBA and/or xSLG.
That’s how this whole thing started, isn’t it? ERA and ERA estimators already know that a high K% and a low BB% are good for pitchers. Both have sizeable correlations to ERA (K% especially) and take up a lot of real estate in estimators such as FIP. However, where these estimators fail and where CRA succeeds is the assumption that results on balls in play should be close to league average. However, we know from some historical examples that this is not always true, and assuming such can be misleading. Here are some examples:
There’s a running joke that “This is the year!” that Chris Archer’s ERA finally regresses to his FIP. The truth is, Chris Archer’s FIP number is consistently bad, as is SIERA, and as is xFIP. From 2016 - 2018, Archer’s ERA has been 4.12, and yet his FIP has been down near 3.65. SIERA is even better. Why? FIP and SIERA assume that a high K, decent GB% pitcher should be getting average results on balls in play, but it’s an incorrect assumption. Archer’s xBA and xSLG were 0.346 and 0.569 respectively last season - both poor numbers - and his 2016 and 2017 marks were just as poor. CRA has Archer at 3.92 for the last 3 seasons, a number I feel is much closer to his true talent level, while some of the estimators that have him as a much, much better pitcher.
2017 Craig Kimbrel - 1.43 ERA / 1.42 FIP / 1.18 SIERA / 2.78 CRA
Compared to its counterparts, CRA didn’t love Kimbrel, certainly not to “historic closer season” levels, anyway. Kimbrel had a whopping 49.6% K% in 2017, to go along with an equally impressive 5.5% BB%, but in the ~45% of PA that didn’t end in a strikeout or walk, Kimbrel’s xBA and xSLG against were .356 and .698, respectively - two huge numbers that likely should not have kept his ERA under 1.50 or maybe even under 2.00. Kimbrel’s 2018 ERA was 2.74.
2016 CC Sabathia - 3.91 ERA / 4.28 ERA / 4.35 SIERA / 3.42 CRA
Even in his Cy Young peak, Sabathia was never a strikeout machine, but from 2015-2017, his K% sat below 20%, a number on the lower side for today’s game. However, CRA sees his .283 xBA and .445 xSLG (both good marks) for 2016 and wasn’t surprised that he had good fortune on batted balls. From 2016 - 2018 (our sample size), Sabathia has posted a 3.76 ERA with the Yankees, and while FIP and SIERA are much higher (both in the 4.30 range), CRA believes the performance to be true (3.61 CRA for those years).
Who CRA Liked in 2018
Noah Syndergaard - 3.03 ERA / 2.80 FIP / 3.21 SIERA / 2.79 CRA
Thor’s K% was the lowest it’s been in his career, but it was still decent nonetheless, and when hitters did make contact, they rarely did any damage, as Syndergaard’s .418 xSLG was the lowest starters mark in all of baseball. Even if that number regresses, it should do so in the form of a higher K%.
Joe Musgrove - 4.06 ERA / 3.59 FIP / 3.93 SIERA / 3.31 CRA
Musgrove is a low K% pitcher at just 20.6%, but he does everything else well; he limits free passes, limits hits, and limits extra bases when the occasional hit does arise. All of BB%, xBA, and xSLG were in the top 20 of starters.
Matthew Boyd - 4.39 ERA / 4.45 FIP / 4.31 SIERA / 3.48 CRA
This is where it’s important to again note that CRA is not built like an ERA predictor but instead an ERA estimator. It does work out that way sometimes, in cases like Sabathia where the weak batted ball profiles carries over from year to year, but that’s not its primary responsibility. In Boyd’s case, with his 2018 batted ball profile (.282 xBA topped all SP), he should have had much better than average results on balls in play and likely did not, skewing his final marks upwards.
Walker Buehler - 2.62 ERA / 3.04 FIP / 3.31 SIERA / 2.72 CRA
Jack Flaherty - 3.34 ERA / 3.86 FIP / 3.57 SIERA / 3.20 CRA
Sean Newcomb - 3.90 ERA / 4.14 FIP / 4.54 SIERA / 3.82 CRA
Who CRA Did Not Like in 2018
Blake Snell - 1.89 ERA / 2.95 FIP / 3.30 SIERA / 3.24 CRA
Obviously, “Doesn’t Like” is being used in relation to his final ERA number here, because a 3.24 pitcher is still a damn good one. Outside of Snell’s 31.6% strikeout rate, nothing he did really stood out to CRA, and he likely benefited from luck and the defense behind him. CRA is in agreement that Snell’s final ERA number should have likely been well over a run higher.
Madison Bumgarner - 3.26 ERA / 3.99 FIP / 4.42 SIERA / 4.42 CRA
Simply put, Bumgarner didn’t do anything particularly well last year, and the 19.8 K% and .543 xSLG on batted balls were on the wrong side of average.
Michael Wacha - 3.20 ERA / 4.22 FIP / 4.60 SIERA / 5.24 CRA
FIP and SIERA think it’s clear that Wacha had some great luck last season in his 15 starts, but CRA thinks he had some absolutely incredible luck based on the quality of contact he allowed. Wacha’s xSLG of .575 was in the worst 20% last year and the .361 xBA was second worst among starters with 15 starts. With sub par K% and BB% to boot, Wacha should have gotten knocked around to a royal degree last season.
Patrick Corbin - 3.15 ERA / 2.47 FIP / 2.91 SIERA / 3.24 CRA
Shane Bieber - 4.55 ERA / 3.23 FIP / 3.45 SIERA / 3.95 CRA
Joey Lucchesi - 4.08 ERA / 4.31 FIP / 3.64 SIERA / 4.51 CRA
Hopefully, I’ll be making improvements to CRA as I begin to better understand how each of its variables interact and what it means to the overall product. The first [next] step is to break down starting innings and relieving innings individually and scale the coefficients accordingly. The fact that the SP IP% and RP IP% is a non-negligible factor in this equation makes me think that the coefficients for our 4 major variables (K%, BB%, xBAcon, xSLGcon) might change a decent amount if we break up our sample into RP innings and SP innings and then merge them, as opposed to how it is currently built, where all IP are measured the same. Stay tuned!
Photo Credit: Jake Roth-USA TODAY Sports