@camarcano
and I proudly present BaseballCV, a repo dedicated to easing the barriers to entry into Baseball & Computer Vision by providing open-source related high-quality datasets and CV models. We're open to additional help, so feel free to reach out!
I’m really excited to announce that I have accepted a position with the Milwaukee Brewers as a Baseball Research & Innovation Intern this summer. Thankful for everyone who has helped me on this journey!
#ThisIsMyCrew
Introducing the next YOLO CV Baseball Model: Pre-Pitch Glove Tracking! Glove location implies an intended location, which leads to an accurate estimate of command by pitch. Special thanks to
@camarcano
for the plate/ball annotations, which allows for a relative miss estimate.
Pitcher-Hitter-Catcher Baseball YOLO Detector Model that is implemented into the 3D Pose Processor in order to ensure proper predictions. Model is a baseline w/o tuning - mAP50-95 for boxes were at a 0.8. OPEN SOURCE, so you can use the weights yourself.
Couldn’t have asked for a better summer working with the
@Brewers
. From making a brief appearance on Draft Day (yellow tie on the right) to being able to contribute to future-oriented high-level projects for a great group of people, this was a dream come true.
Going open source! The YOLO Bat Prediction Model and documentation are now completely available on my Github! May require some technical know-how to utilize the model right now, although I am working on an app to overcome this!
Experimenting with a new metric, pSERA (Predictive Stuff ERA), in part of research being conducted for
@ProspectsLive
. Preliminary results are extremely promising against the most commonly used predictive metrics, as shown. Will continue to be tweaked.
Finalized pSERA / pSTFERA suite results are out in the article! In specifically looking at pSERA, it was able to out-predict any ERA estimator on Fangraphs. Will try to get scores out for all of the apps as soon as possible.
Introducing another
@DrummeyDylan
statistic, pSTFERA.
pSTFERA is the group of predictive pitch-level metrics that integrates pitch modeling to try and more accurately predict future performance based on current performance.
pSERA (Predictive Stuff ERA), pLERA (Predictive
The Bat Tracking Model has upgraded! Using 6,956 train and 1,740 test images, the model has a 73.2% Pose Precision, 86.7% Recall and 75.4% mAP50 (huge improvement from last model). Also added line tracking visuals and hope to get a slight improvement before getting metrics.
First Post Back! This time, using CV to estimate 3D points on a pitcher. Messed with some demo code from MMPose for this Tim Mayza v Luis Rengifo pitch from 2023, which utilizes a detector and 2D pose estimator going into a 3D Pose Estimator.
Ethan Small had the best pitch in terms of Arsenal+ (min 200 thrown) in all of AAA last season. Here's the new and improved MiLB Pitch Card showcasing the pitch.
Small preview of something being worked on in my new role as
@PitcherList
Assistant Director of Research and Analytics under
@blandalytics
with PL-based Pitch Cards (source code via Kyle). Definitely still needs to be tinkered with, but going to be a great tool once available.
More CV Bat Tracking in YOLO! The primary differences in this version were more epochs and sample size. ~3600 images and ~600 instances were trained for 75 Epochs. Currently at a .546 Precision and .622 Recall for Pose Modeling. Definitely not perfect, but a vast improvement.
Stuff-Based Arsenal+ Leaderboard for D1 College Baseball through 4/1 TM Verified Games. To qualify, pitchers must’ve had 500+ and teams 2000+ pitches tracked.
Experimenting with a Bat Identification Model using YOLO5 to hopefully get Swing Metrics down the line. Performance improved from v1, but clear that training set needs to be larger (currently around 200 images) for enhanced accuracy. 4K Test Video from
Made a much more visually appealing version of the Baseball 3D Pose Estimation using RTM models as the detector and 2D Pose Estimator with MotionBERT serving as the 3D Estimator. Also wanted to track predictions, so adding functionality for that.
Major progress in the YOLO Bat Tracking Model! Was able to implement Speed Estimation through line points to get some preliminary estimates of Bat Speed. Don’t believe they are the most accurate figures, but this shows some type of estimate is possible.
Arsenal+ scores will be out on the AAA Stuff App on
@ProspectsLive
within the week. The results shown in the article were extremely promising concerning the metrics predictive power compared to other Stuff metrics - already working on improving it further.
Introducing 𝗔𝗿𝘀𝗲𝗻𝗮𝗹+, a metric created by
@DrummeyDylan
that considers the stuff and location of a given pitch within a pitch type, as well as consideration of the prior pitch type and the difference in these numbers.
In terms of my AAA Stuff+, Cooper Criswell had the best slider (including variants) in the league with a 118 (min 200p). Pictured here is his Sweeper's
@PitcherList
Pitch Card, which now includes Spin Efficiency (thanks to
@pobguy
's work). The next update incoming...
More Bat Modeling! In V2, the model was upgraded to YOLO8, had about 3x the sample, and utilized a pose base over a rectangle. This predicted video (via MLBAM) utilized a minimum 50% confidence for detection which led to some gaps. More on noticed struggles below🧵
More Baseball Pose Estimation! Exploring some custom solutions utilizing the YOLOv8x-Pose as the 2D Estimator and applying some basic depth estimations (as baseline). Performance of the 2D points is lacking, even with a bigger YOLO version - will likely go with another model.
Leaning on Deferred Contracts was one of my major recs for Large Market teams to optimize spending (written in September for
@PitcherList
). For a refresher on navigating contract spending and the CBT, check the entire piece out:
Updated my Savant Video Scraper (to help with CV projects) to allow for the user to either go off a CSV or pull Play IDs from Savant within certain dates, with the option to specify a team and/or pitch call. Feel free to check out the repo here:
My 2023 AAA Stuff / Location / Pitching+ App is Out! Explanation behind the models will be elaborated upon at
@ProspectsLive
soon… For reference, models are count-adjusted, pitch-adjusted, and based on xRV. Also, App is better on Desktop vs Mobile.
With stuff models needing to adjust by weather factors, here’s the script I used to get data from
@meteost
based on the nearest weather station for each stadium. Planning on adjusting in the future for given schedules / requests.
(1/2) Environmental factors have a known effect on Stuff - its inclusion is being incorporated into my models. I've compiled 2021-2023 Daily Weather Data based on Stadium location for anyone interested in including these factors in a model via MeteoStat.
If you wanted to utilize the YOLO Bat Prediction Model for MLB Swing Video but didn't want to program, now you can! I made a basic Streamlit App to allow the user to upload a video and download the predicted clip (Allow 5-10 Mins for Preds). Check it out:
I was able to get the scores for my new metric, Arsenal+, updated for the AAA Stuff App on
@ProspectsLive
. Stuff metrics are also on V2 and moved to the RE288 basis, so some scores may vary from V1. RV/100 is also now included in the app.
Since the goal of the YOLO Models is intended to go beyond video, I’ve included some data from the model’s .txt predictions for Glove Tracking (Class 0 = Glove). Based on preliminary estimates, this pitch missed its target by ~22 inches. Still tweaking, but progress.
Not really sure what models are classifying his fastball as only have 101 Stuff, but my models absolutely love it - 114 Stuff+, 110 Pitching+, 109 Arsenal+. Also has 7 FT extension, so perceived velocity is a bit higher than 95.
DL Hall’s fastball seems pretty ordinary. 101 Stuff+/15” IVB/95mph.
But BA gives the pitch an 80-grade. Why?
Well, Hall’s FF holds a -4.2 VAA at a 5.3 V-Rel with excellent command. 15HH%/.240xwOBA/30Wff% and plays incredibly well up in the zone.
Elite.
The AAA Pitch Modeling App for
@ProspectsLive
is back for the 2024 season! New Pitching and Modeling data for Minor League Players is now available by tapping on the “Switch to 2024” button on the app.
Our newest staff member
@DrummeyDylan
makes a splash in his debut article, releasing his version of Triple-A Stuff+ exclusive for Prospects Live.
It automatically trains for optimized model types and hyperparameters to ensure the best predictive measures for better performance.
(1/🧵) Breaking down the process of YOLO Bat Tracking Model, here is its current model architecture. Model trained on baseline YOLOv8l-Pose and uses an array of convolutional, SPPF, upsampling and other layers to process multiple types of bats and camera angles.
The Bat Tracking Model has upgraded! Using 6,956 train and 1,740 test images, the model has a 73.2% Pose Precision, 86.7% Recall and 75.4% mAP50 (huge improvement from last model). Also added line tracking visuals and hope to get a slight improvement before getting metrics.
More Baseball Data Tools! Amateur Baseball / Showcase data is generally not very friendly to those attempting to compile lots of info for rankings. To combat that, I created a repo to scrape Event Data and Player Data from Perfect Game into CSVs.
Looking into Sequencing at the 2023 AAA level for
@ProspectsLive
and given sequences that happened min. 1000 times, these are the average run values. Worth noting that these are raw numbers - no adjustments were made, although future player eval tools will be adjusted.
NCAA D1 Arsenal+ Leaders (min 150P) through games played this weekend (3/10). If you’re not familiar with Arsenal+, you can check out the explanation for this new Stuff-type model I wrote for
@ProspectsLive
here:
FSL & AAA Stuff Scores are now updated through 4/29 on the
@ProspectsLive
Leaderboard. Last Update for a bit (Finals Week, etc.), but I’m planning on adding Sequencing+ & pSTFERA scores as well as some potential new metrics over the next couple of months.
The lack of a Public Leaderboard currently is a big reason behind my work on utilizing CV to get granular bat data. Even when the metrics become public, the ability to visually track a bat using common game film has inherent value.
Unfortunately no public leaderboards as bat tracking data isn't publicly available yet.
(Hopefully soon
@mike_petriello
👀)
In the mean time here's the top 25 from 2023 per
@DrivelineBB
model
New Stuff-Based Arsenal+ Leaderboard for D1 Baseball Players through 4/8 TM Verified Games. To qualify, pitchers must’ve had 600+ and teams 3000+ pitches tracked. If you're not familiar with Arsenal+, check out the metric here:
All Pitch / Pitcher Grades for Stuff Scores have been updated through 4/15 for both AAA and FSL on the
@ProspectsLive
Minor League Stuff App. If you’re not familiar, check it out:
While the model uses a human detector before extracting key points, some predictions would switch from person to person. Training a detector model so that pose predictions are made within a given detected box. This also allows labeling, so Person 2 becomes Pitcher, etc.
Made a much more visually appealing version of the Baseball 3D Pose Estimation using RTM models as the detector and 2D Pose Estimator with MotionBERT serving as the 3D Estimator. Also wanted to track predictions, so adding functionality for that.
@tangotiger
Hitter aspect is fascinating, but personally, my attempt to solve it through CV was trying to answer pitcher questions. Deltas from batter averages in Bat Speed, Swing Length, and other derived metrics between pitchers seems promising.
A Brief Preview of something being worked on for
@ProspectsLive
in an attempt to better understand pitching. All models (including the prior Stuff+ ones) are moving to an RE288 basis, and these are the corresponding R2 values as the sample requirement of a given pitch increases.
For everyone asking, Major League pSTFERA Suite scores are out at the link below! There will be updates and potentially a total metric, but this is a good start. Excited to see how well these numbers perform in predicting 2024 results.
Sequencing+ scores for will be updated on the AAA app soon. Should be treated as supplementary metric - Sequencing isn’t as predictive as stuff. But, the scores do explain variance in actual v expected performance. Stuff sequenced well is more predictive than stuff alone.
The best two-pitch sequence to open an at-bat(according to Triple-A expected Run Value)?
A curveball followed by a slider.
@DrummeyDylan
is back with another metric called Sequencing+ that measures the expected run value of a given PA based on the sequence of pitch choices.
had the fortunate time to catch
@GreenfieldMax18
@CoachB_Smith
@DrummeyDylan
@drivelinekyle
on pitchcon tonight about pitching development’s current status and some of the management tactics they use at their colleges or as coaches in general. thank you guys!
Our new MLB STUFF APP is Out! Stuff+, Location+, Pitching+, Arsenal+, and Sequencing+ scores are all available - each model has been researched and tested publicly. The app is linked below:
Introducing Arsenal+: A Stuff Metric Considering How Pitches Play Off One Another (4/6):
Dylan (
@DrummeyDylan
) is super smart and created a new pitching model (Arsenal+). He dug into stuff and location of a given pitch type and also incorporated prior pitch type. 🤯
(2/2) To that end, in theory, these models serve as a better descriptive gauge versus predictive.
@tangotiger
clarified the need for tailoring to individuals in a predictive model, which Location+ does not - it’s generalizes.
One month until the college baseball season! Our guys at On The Clock break down our recent top 100 MLB Draft Boards.
Then, Jared, Tyler, & Dylan are joined by Kentucky OFer Ryan Waldschmidt (
#28
on our list!)
Apple:
YouTube:
(1/2) Very excited to be mentioned by
@bymichaelrosen
in his piece about measuring 4-Seam Command w/ The Kirby Index. Great preview of solving Command from an evaluation perspective. To that end, I believe CV methods offer a better long-term solution in Player Dev.
@QuinnRileyBB
@Drew_Haugen
Obviously an awesome chart, but I think it’s worth reiterating what
@DolphHauldhagen
said about the importance of additional context with IVB. A pitch can still be above-average w/ average IVB depending on release/VAA. Still interesting, but it may be worth going deeper.
He may throw fast, but his slider was the headline - 46% CSW on the night. Contact numbers are admittedly worrisome, but small sample. Interested to see how it plays out.
Rookie Mason Miller got robbed of a no-no last night after the A’s pulled him after 7, but the future is bright for the 24 yo righty ⭐️
In 15.1 IP, Miller ranks towards the top in:
-fastball velocity (94th percentile)
-barrel % (90th percentile)
-fastball spin (86th percentile)
@dgerth1305
Usually prefer Jupyter, but with this, I used Colab so I could have access to a solid GPU. Original CPU Training Times were >1H 15M vs the A100 GPU which took 10 minutes tops.
Thank you!!
I absolutely will. I somewhat mentioned it later in the article, but I was debating about making the metric FIP-based in an attempt to be more predictive in general. Tested some rough adjustments and will further investigate.
Terrific stuff Dylan!
And yes, that location-metric is what we'd expect as it provides such little signal for the future. It's a classic "overfit" kind of metric.
Can you also include Classic FIP in the future please?
Just because teams are throwing fewer four-seamers, doesn't mean the fastball is dead. On how the game might be changing its approach to the most common pitch type:
Original pSERA results did not play as well as hoped over multiple years - with some restructuring, I believe the metric to be more sound despite slightly higher RMSE. Results again strong, especially between 60-120 IP minimum thresholds. Great for Starters and High-IP Relievers.
Phoenix Sanders (
@PSanders_14
) was the Best Sequencer in AAA last season (min 100 BF) with a 136.7 Sequencing+. Plus, he has 3 above-average Pitches via Stuff Models. Leans heavily on a slider that had a 113/108 Stuff+ in Norfolk/ Sacramento. Great candidate for any MLB org.
🚨Pro Day Update🚨
As a reminder to any organization that has not been contacted - our pro day is January 12th - 14th.
We welcome any MLB, Indy, or Foreign Organization.
Please email ross
@treadathletics
.com for more info!
Tentative schedule ⬇️
To add on to the Leaderboards posted earlier, these staffs led D1 Baseball in l stuff-based Arsenal+ through 3/18 TM Verified Games (min 1500p under system).
One of my favorite pieces to write - the way people evaluate contracts needs to be based on more objective reasoning. The whole topic deserves a paper… I tried to provide a briefer explanation for the sake of simple understanding.
Quick Preview of the top MLB scores for a new metric being worked on for
@ProspectsLive
- Sequencing+. Sequencing+ isolates the most successful sequencing strategies while adjusting based on certain PA-level factors. A full introduction will be posted soon.
Somewhat odd. My AAA Stuff Models on
@ProspectsLive
liked Kyle Hurt as a above-average arm. Location was a 108, Arsenal+ was a 106 compared to a 101 Stuff+. His pitches and locations seem to play very well off one another, which might be what the other models are missing…
@enosarris
Any thoughts on why your model "hates" Kyle Hurt? MLB sample is super small, but you have him as 89 Stuff+ in AAA. FF only had 14.6 iVB there, so might explain it, but results were really good.
@LoyalRo29280019
@ProspectsLive
They’re close - most predictive models are playing at the margins in general. From the look at 2021-2023, I’d put xFIP and SIERA above xERA and botERA. The pSERA managed to seperate itself by a good deal.
I feel this misleads. To calculate the importance of the “big inning”, don’t do (W | SBI * PBI) - that’s just more representative of outscoring opponents in general. Look at them separately w/ (W | SBI) & ( W | PBI) to compare the relative importance of these types of innings.
How important is the 'big inning?'
Teams who generated a big inning (3+ runs) and prevented their opponent from having a big inning in a given game have won...
95% of the time (D1 Softball)
92% of the time (D1 Baseball)
@ajaypatel8_
I looked into this with Arsenal+ somewhat… taking the difference of a given subset of characteristics as well as the actual pitch characteristics. Results were promising - including location hurts the stickiness a bit.
@drivelinekyle
Since Driveline is beginning to add CV modeling to the OBP, is there any plan for Bat Identification Modeling? Or at least a collection of training images? I’ve begun to dig these up myself, but didn’t know if additional resources would be available.
For a lot of amateur sabermetrics fans, spitting out other arguments is the only way one can actually support their cases. What separates the Amateurs and the Knowledgeable is how they approach the pre-researched answer or new problem:
A THREAD🧵:
Of course that's your contention. You just learned about baseball analytics for the first time. You just finished reading about WAR in Fangraphs' glossary for the first time and are convinced that pitchers can't control batted balls and
@BeRett21
Ideally, I’d like to get similar to Statcast-like metrics from the tracking. From there, visual tracking could potentially model the swings themselves.
Had this series in the works for a while, and I’m glad it’s finally public! In a sport with constrained resources, finding the best ways to use those resources to maximize team success is crucial, which is what I investigate through the framework of spending optimization.
@ajaypatel8_
Absolutely - the main goal was to see if it could further explain why certain players consistently outperformed their stuff metrics (also works in inverse). Gave it a whole write up - hopefully it adds to what you’re looking at.
@Johnny_Davis12
Is Count factored into the model at all and/or adjusted? I’ve generally tried to factor it in for my AAA Stuff iterations just due to the overall run values and the results you’re seeing.
@BenVerlander
MLB matchups need the highest sample for true outcomes - competitive balance has nothing to do with number of championships. SD of W%, Run Differentials, and # of Blowouts all do. By these measures, the league is less balanced. These were my results:
@drivelinekyle
@clayton_t22
Have you guys tried any partially automated labeling solutions? I’ve had some success using prior models to predict labels with a high threshold (p>.8-.9) and then manually checking to ensure accuracy.
New Week means new Leaderboards for modeling scores of D1 Stuff-Based Arsenal+ (grades through 3/25 TM Verified Games)! Minimum Thresholds are 1500p tracked for Teams and 300p for Players. Check out this piece if interested in learning about Arsenal+:
Love it! I think if you’re actually trying to rank players, Pitching+ would be more useful just as it both stuff and location together, predicting xRV and therefore articulating overall value better. From a Dev perspective, identifying Loc & Stuff separately is preferable.
This is super useful data. Best Storm Chasers, by Stuff+ last season (my hype around Hoffmann and Coleman gets evidence):
Coleman: 112.57
Pennington: 109.37
Dipoto: 107.47
Klein: 107.22
Hoffmann: 102.07
Heasley: 99.91
Bowlan: 99.18
Castillo: 94.49
Veneziano: 94.02
Parrish: 84.09
The Dunning-Kruger effect has some clear applications to sabermetrics. The “peak of ignorance” came on when those old-school claims were dismissed when those claims weren’t ready to be rejected. Willingness to accept the old with new evidence shows we’re on the upward slope.
@enosarris
The more cool stuff that's uncovered, the more it seems like old-school thoughts we dismissed actually have (some) merit.
"You have to be able to locate your fastball!"
Added to this! The repo now has additional player scraping functions to access data more easily. 1. A Filter that can utilize Grad Year, Position, and Age to scrape only those players. 2. A name filter, which allows “First Last” to be typed to directly get data.
More Baseball Data Tools! Amateur Baseball / Showcase data is generally not very friendly to those attempting to compile lots of info for rankings. To combat that, I created a repo to scrape Event Data and Player Data from Perfect Game into CSVs.
Check out the newest article on the blog!
Jackson gives an introduction to analyzing a baseball savant page and some of the stats you will encounter.
Article link: