I’m thrilled to announce that I’ve been awarded an
#ERC
Consolidator Grant. 2 million Euros to work on what I call „Deep Learning 2.0“. I hope that the next generation of DL can achieve Trustworthy AI by design. Details about DL 2.0 in this blog post: 1/3
This may revolutionize data science: we introduce TabPFN, a new tabular data classification method that takes 1 second & yields SOTA performance (better than hyperparameter-optimized gradient boosting in 1h). Current limits: up to 1k data points, 100 features, 10 classes. 🧵1/6
Hooray, we got 5/5 submissions accepted at
#ICLR2022
. Congratulations to all my wonderful students and collaborators. A brief overview of our papers below:
I like this new paper on DL vs trees at the
#NeurIPS
2023 datasets and benchmark track: . One of their findings:
#TabPFN
performs well across 98 datasets: it ties for 1st place with CatBoost in predictive performance but is 200x faster to train. This is
I like this post and would advise all PhD students to follow it! In a dynamic field like machine learning, life always keeps on getting busier (even once you have a tenured position and brilliant PhD students to keep up with ;-) and you need to schedule breaks to recharge!
I'll relax once I submit this paper.
I'll relax once I submit my thesis.
I'll relax once I defend my thesis.
I'll relax after I get a post-doc.
Damn! I guess I'll have to relax after I get a permanent position.
No. The future always gets busier.
Schedule proper breaks NOW. 💛
Please see our blog post for details & code. Also, we just got an oral at the
#NeurIPS
table representation learning workshop 🎉Joint work with my outstanding students
@noahholl
,
@SamuelMullr
and
@KEggensperger
. 6/6
TabPFN is radically different from previous ML methods. It is meta-learned to approximate Bayesian inference with a prior based on principles of causality and simplicity. Here‘s a qualitative comparison to some sklearn classifiers, showing very smooth uncertainty estimates. 2/6
Thanks for the many comments on my tweet on TabPFNs:
Here's an update on how this may revolutionize data science.
TL;DR: learning classification algorithms rather than programming them; real-time AutoML; democratizing ML to tackle *small data*! 🧵1/9
This may revolutionize data science: we introduce TabPFN, a new tabular data classification method that takes 1 second & yields SOTA performance (better than hyperparameter-optimized gradient boosting in 1h). Current limits: up to 1k data points, 100 features, 10 classes. 🧵1/6
Exciting news: following a series of 8 AutoML workshops at ICML, we will hold the 1st International Conference on AutoML in 2022. Co-located with ICML, 3 days, currently planned in person. Website: Submission deadline: Feb 24, 2022
#AutoML
#AutoML_Conf
1/
6 accepted papers at NeurIPS. Amazing! 4 at the main track and 2 more at the benchmarks and datasets track. Huge congratulations to my bright and hard-working students & collaborators, they couldn't deserve it more!! Previews to early versions in this thread, more to follow soon.
Deep Learning, in particular even simple MLPs, can outperform gradient boosting for tabular data, if you use the right „cocktail“ of different modern DL regularization methods. Go
#AutoML
! We tried different GBDT implementations, hyperparameter spaces, etc, with similar results.
Regularization is all you Need: Simple Neural Nets can Excel on Tabular Data
pdf:
abs:
well-regularized plain MLPs significantly outperform sota specialized neural network architectures
We got Transformers to do basically perfect Bayesian inference with any prior we can sample from. Super cheap (in a single forward prop) & more accurate than MCMC or VI. This could be really disruptive. , with
@SamuelMullr
,
@noahholl
, Sebastian, Josif
The
#ICML2023
rebuttal deadline was Sunday. That’s sad from a diversity & inclusion point of view; there goes the weekend with the family. For , we have “no deadlines on weekends” in our internal guidelines. I hope
@icmlconf
will follow suit in the future.
TabPFN happens to be a transformer, but this is not your usual trees vs nets battle. Given a new data set, there is no costly gradient-based training. Rather, it’s a single forward pass of a fixed network: you feed in (Xtrain, ytrain, Xtest); the network outputs p(y_test). 3/6
🎉 I’m delighted that my amazing team got 6
#AutoML
papers accepted to
#NeurIPS2023
, including an oral on AutoML for Trustworthy AI! Other topics include
#NAS
,
#BO
, practical
#HPO
, LC extrapolation and
#LLMs
for data science. Congratulations to all my coauthors! Details below🧵1/
Time for an update on experiments with our learned TabPFN: we performed a much more comprehensive analysis on 179 datasets, confirming the patterns from our previous evaluation on 30 datasets and looking in much more detail. A long thread with lots of plots🧵1/17
Imagine the possibilities! Portable real-time ML with a single forward pass of a medium-sized neural net (25M parameters). Go,
#GreenAutoML
! Please share widely; there are endless possibilities for improvements & extensions and we'd love to tackle them with your help. 5/6
TabPFN is fully learned: We only specified the task (strong predictions in a single forward pass, for millions of synthetic datasets) but not *how* it should be solved. Still, TabPFN outperforms decades worth of manually-created algorithms. A big step up in learning to learn. 4/6
🏆 Honored to receive the
#KDD2023
Test of Time Award for our Auto-WEKA paper from 2013, together with Chris Thornton,
@HolgerHoos
and
@k_leyton_brown
. A decade later, it's humbling to see the lasting impact of our work in the ever-evolving
#AutoML
landscape.
I'm very grateful
#ICLR2024
released reviews to authors Friday evening at 7pm European time. Have a fun weekend, poor grad students 😕 And even worse, poor parents! Not a good move,
#ICLR
; so available by conscious scheduling! Let’s hope
#ICML2024
and
#NeurIPS2024
will avoid this.
This is a good example of interaction effects between hyperparameters. The original weight decay value was optimal before adding various other regularization methods; but, after adding these, decreasing weight decay from 1e-4 to 4e-5 gave the single most important improvement!
The Deep Learning Devil is in the Details. I love this work from
@IrwanBello
and collaborators in which they show how training "tricks" improve ~3% absolute accuracy on ImageNet, progress equivalent to years of developments and research!
Paper:
If you’d like to do a PhD in
#AutoML
or
#MetaLearning
in my group at the
#ELLIS
unit Freiburg, or in another ELLIS unit co-supervised by me, please apply to the ELLIS PhD program and indicate your interest in working with me:
The deadline is December 1st!
At
#AutoML_Conf
, we care about
#reproducibility
. Today, we're introducing the novel concept of reproducibility reviewers, who assess to which degree results can be reproduced with the shared code. Full story:
Sign up as a reviewer:
Congrats to the TorchVision team, who just trained the best-ever ResNet using our TrivialAugment () and CosineAnnealing () methods (amongst several other recent DL training tricks):
Here are all the infos for
#AutoML23
in one place. Please share widely -- it's going to be great!🚀
Call for Participation:
#AutoML
conference 2023, Potsdam/Berlin, September 12-15, 2023.
Some reasons to join the conference:
- 5 great on-site keynote
My team and I are at NeurIPS, presenting a total of 17 (!) papers: 4 in the main proceedings and 13 at workshops.
I'm very much looking forward to meeting old friends in person again, and also to making new ones!
Here is a quick overview of our papers (short tread of 6 posts):
Do you like
#LLMs
and
#AutoML
? Then you might like the Auto-Survey competition at
#AutoML2023
(), which just launched officially. It's a very novel competition where LLM agents write survey papers and LLM agents judge them (final judges are human). IMO
Surprising turn in
#HPO
and
#NAS
: we now have DEHB, an evolutionary successor of BOHB that's up to 1000x faster than random search for optimizing 6 NN hypers and up to 32x faster than BOHB on NAS-Bench-B201!
#IJCAI21
paper: Code:
Congratulations to my fantastic MSc student Shuhei Watanabe, for winning the multi-objective hyperparameter optimization competition for transformers at
#AutoML_Conf
!
We extend Bayesian optimization (BO) by user priors over promising regions of the search space. This expert-guided BO is up to 12x faster than vanilla BO and allows DL experts with little compute to embrace BO. , with Carl, Danny,
@LindauerMarius
,
@luiginardi
#GPT
meets
#AutoML
: in an effort to integrate user knowledge into AutoML, our new tool CAAFE uses LLMs to generate semantically meaningful features for tabular data (and also explains them). Towards an AI assistant for human data scientists🚀
Paper
Demo
Fantastic news, I’m honoured and humbled to receive the AIJ Prominent Paper Award 2021! Super happy to receive this award with this amazing set of coauthors,
@HolgerHoos
,
@k_leyton_brown
& Lin Xu, that I started my research journey with and that formed me so very much! Thanks!
Very proud and humbled to have won, together with
@k_leyton_brown
and our former students,
@FrankRHutter
and Lin Xu, the AIJ Prominent Paper Award 2021 (as just announced at
@IJCAIconf
), for our work on using
#MachineLearning
for predicting running times of
#AI
algorithms.
I'm at
#NeurIPS2023
with my amazing team, excited to be presenting 6 papers at the main track and 2 at workshops, as well as a keynote in the table representation learning workshop. Here's all the info in one tweet, ordered by day. Please come by and chat with us 🙂
Tuesday
Excited to present ELIZA, the Konrad Zuse School of Excellence in Learning and Intelligent Systems, a new AI Lighthouse comprising 7 German ELLIS Units, to attract the brightest MSc & PhD students to Germany. 1/2
Many DL papers for tabular data didn’t deliver, so I understand the skeptics. But I’m a champion for reproducibility and hope to convince you that way. All code is available at with demos. Here is a Colab comparing to baselines: 9/9
For everyone submitting
#NAS
papers to
#NeurIPS2021
: I'd encourage using the NAS best practices checklist () to ensure your work is reproducible. Here is a .tex file you can include in your appendix next to the NeurIPS checklist:
#AutoML_Conf
2022 is now open for submissions!
Abstract submission deadline: February 24th
Full paper submission deadline: March 3th
Please see the call for papers and the author guidelines:
I have 2 new
#AutoML
PhD openings in our new Collaborative Research Center (Sonderforschungsbereich) „Small Data“, a major collaboration between the Freiburg university clinic and AI folks, with ~30 PhD students.
1st application deadline: June 18
1/4
Announcing the late-breaking workshop track at
#AutoML_Conf
: paper submission by April 20, notification May 18
These short workshop papers are compatible with submission of full versions to a conference, e.g.
#NeurIPS
. Looking forward to great papers! 1/2
I'm thrilled to announce that our proposal to run the first
#NAS
workshop at
#ICLR2020
was accepted. We're looking forward to building a vibrant, open, collaborative, diverse and welcoming community around neural architecture search! Prepare to submit your work in late January!
Congratulations, Dr
@AndreBiedenkapp
to finishing your outstanding thesis on
#DAC
with summa cum laude, the best possible grade! It’s been a huge pleasure to supervise you over the last years, and to have you as a positive force & integral part of the lab. Thanks, Dr. Andre!
Congratulations to Dr André
@AndreBiedenkapp
. It was my greatest pleasure to co-supervise you for the last years. You showed an outstanding performance!
At the AI Safety Summit in London many risks will be discussed, but the biggest risk for Europe is different: the LLM ship has sailed without us, and we are on our way to an all-encompassing dependency on American companies.
Europe needs to wake up and invest heavily in AI. We
Good morning Schluchsee! What a beautiful location for an
#AutoML
research retreat with my amazing group. The perfect setup for planning the next big thing in AutoML 🚀
Excited to introduce the first large-scale efficient-to-evaluate NAS benchmark, NAS-Bench-301: Exactly for the DARTS space (10^18 architectures)! Enabled by a surrogate model with lower estimation error than the noise in a single SGD run in NAS-Bench-101.
I recently had the privilege to work with fairness-aware ML folks to explore the question Can fairness be automated? Spoiler alert: No it can’t.
#AutoML
needs to be extremely cautious in this domain but still has an important role to play.
This was an amazing project that left
We
#metalearn
proxy MDPs called "synthetic environments" for RL: given an RL environment E, you can train a neural net to replace E‘s original MDP & train policies for the original E more efficiently! , with
@FerreiraFabioDE
, Thomas N.,
@AndreasSalinger
@predict_addict
Dear Valeriy, this might be hard for you to accept: an independent evaluation on 98 datasets at the
#NeurIPS2023
datasets and benchmark track () showed TabPFN to tie for 1st place with CatBoost in predictive performance but to be 200x faster to train. This
📢To not follow CHI and CVPR as
#COVID
super spreader events, with all organizers and senior area chairs we decided that
#AutoML_Conf
will mandate vaccinations, daily antigen self-tests, and masks inside. Hoping that our attendees safely make it through the preceding
#ICML2022
...
More exciting news about
#AutoML_Conf
: we will have a journal track: , chaired by
@HolgerHoos
. In-scope open access papers recently published at top-tier journals will get the opportunity to present at the conference exactly like regular papers. 1/
So excited that our work on TabPFN received the best paper award at the NeurIPS Table Representation Learning Workshop on Friday,
#neurips
#TrlWorkshop
🥳🥳
This workshop has been so exciting, merging perspectives for semantic and more classical tabular data learning!
This
#NeurIPS2023
paper presents a really simple and effective idea: use LLMs to understand the context of a tabular data set and write code to engineer new features for it. Moving
#AutoML
more towards automated data science! Congratulations,
@noahholl
and
@SamuelMullr
!
Excited to present "LLMs for Automated Data Science: Introducing CAAFE for Context-Aware Automated Feature Engineering" at
#NeurIPS2023
!
We had a simple idea: Can we use the domain knowledge of LLMs to automate the feature engineering process?
with
@SamuelMullr
@FrankRHutter
Our Springer book on Automated Machine Learning has finally been published :-) Edited together with
@larskotthoff
and
@joavanschoren
Get the free PDF or the hard cover:
Also see the book website:
#AutoML
Together with 3 Turing Award winners (
@geoffreyhinton
, Yoshua Bengio, Andrew Yao), Nobel laureate
@kahneman_daniel
, Stuart Russel and 20 other AI & policy leaders, we just released a commentary that recommends actionable measures towards making AI safer:
Great to have the next generation of Auto-sklearn: Lots of improvements, including multi-fidelity optimization for fast progress on large datasets and theoretically-grounded meta-learning across datasets to find a strong set of configurations to start from
Excited to announce Auto-sklearn 2.0!
Introduces a lot of improvements for large datasets, portfolio-based warmstarting, and meta-learning to set its own hyperparameters.
Blog:
with
@KEggensperger
, Stefan Falkner,
@LindauerMarius
and
@FrankRHutter
For everyone submitting
#NAS
papers to
#ICLR2021
: I'd encourage using the NAS best practices checklist () for your reproducibility statement. Here is a .tex snippet you can fill out and include:
Really looking forward to
#AutoML_Conf
: Next to 6 great keynote speakers, we'll have 5 tutorials, panels on AutoML & fairness and AutoML & large pretrained models, 4 poster sessions and lots of social/networking time, including a banquet on a yacht! 🧵 1/
Our
#NeurIPS
2022 workshop on
#metalearning
() will be hybrid. We also strongly encourage submissions from authors who do not plan to attend NeurIPS in person. There will be a virtual poster session. 4 pages + unlimited refs & appendix. Deadline Oct 1 AOE
In our
#NeurIPS2023
oral „Rethinking Bias Mitigation: Fairer Architectures Make for Fairer Face Recognition“, we use
#AutoML
for Trustworthy AI!
More precisely, we use multi-objective joint
#NAS
&
#HPO
for fairness mitigation in face recognition.
The main conclusion:
While I'm thrilled for the authors of accepted papers I feel with the authors of the many rejected papers. Don't let this discourage you! Rejects happen to all of us; I had 6/6 rejects once, there is a lot of noise in reviewing. What helps is using criticism well & perseverance.
I'm happy to announce 4 competitions affiliated with
#AutoML_Conf2022
: Zero-cost NAS (), multi-objective HPO for transformers (), DAC4AutoML (), and meta-learning from learning curves ()
I’m not saying that our particular trained TabPFN revolutionizes data science. It totally doesn’t. But learning classification algorithms rather than programming them is pretty wild, and I do believe this is where the future of ML lies. AI-GAs
@jeffclune
6/9
What an amazing year this was. Next to the great advances in AI we also have a home conference for
#AutoML
now! Thanks to all who helped make this happen
@automl_conf
! And thanks to my awesome students for a great year. Let's take time to recharge batteries now🙂Happy holidays!
We show that NAS-Bench-101 and -201 are not enough and release a unified
#NAS
benchmark suite with 25 benchmarks. A unified interface allows reliable benchmarking of NAS methods. , with
@y_mehtu
,
@crwhite_ml
,
@ZelaArber
,Arjun,Guri, Shakiba,Mahmoud & Kaicheng
This week my fantastic PhD student
@KEggensperger
defended her PhD on
#AutoML
,
#HPO
and
#benchmarking
, with summa cum laude (perfect grade). It's been a privilege to be your supervisor, Katha! Katha is on the market now; be sure to make her an offer that's as awesome as she is!
Our
#AutoML
MOOC on the AI Campus has over 1000 attendees as of today :-)
It’s entirely free, including the certificates. I’m very happy to help shape the next generation of
#AutoML
experts :-)
With
@LindauerMarius
and
@BBischl
#KICampus
As a diversity & outreach measure, I'm thrilled that the first 100 online student tickets for
#AutoML23
are free: Please register while tickets last & spread the word!
We also embrace hybrid participation this year: even the poster sessions are hybrid!
Congratulations to my team,
@SimonSchrodi
,
@crwhite_ml
, Ekrem Öztürk & Danny Stoll for winning 2nd place in the first
#NAS
competition! Especially impressive seeing that, contrary to the 1st and 3rd place, in the spirit of the rules, we refrained from using data augmentation.
While I’m thrilled with the 100% acceptance rate of our
#ICLR2022
submissions, I feel with the many authors who were less lucky. I’ve been there, too, many times. Peer review is quite noisy, but what pays off is perseverance and the ability to accept & incorporate feedback.
By popular demand we extended the deadline for the
#MetaLearning
workshop at
#NeurIPS2021
to September 29th. This is the day after the
#ICLR2022
abstract deadline, so you can submit as a milestone for polishing up your ICLR submissions.
4/4 papers accepted at
#ICLR2020
, including an oral and a spotlight! What a nice Christmas present to my hard-working students :-) They deserve it! Thanks to the PCs, ACs, and reviewers for all their hard work with the reviewing process!
DL 2.0 puts the domain expert back into the picture to specify (multiple) objective functions that matter in a particular application. Then multi-objective AutoML (optionally guided by DL experts) finds Pareto-optimal architectures, optimizers, SSL pipelines, hyperparams, etc 2/3
Great news, our
#ICLR2020
workshop on
#nas
got accepted! Deadline will be late February; please consider the NAS best practice checklist from our JMLR paper: . Organizing together with Aaron Klein, Jan Hendrik Metzen, Liam Li, Jovita Lukasik, and
@ZelaArber
We're starting a series of blog posts about our 7 neural architecture search (NAS) papers from 2019: First up: my PhD student Thomas Elsken about our ICLR 2019 paper on truly multiobjective NAS (experiments with up to 5 objectives):
Would your company like to connect to the
#AutoML
research community and/or recruit in this field? If so, you might be interested in the call for sponsors for
#AutoML_Conf2022
:
Another way in which this may revolutionize data science is by comparison to AutoML systems. I’m the senior author of Auto-sklearn, a very successful 7-year project with 79 contributors. TabPFN was learned in 20h and in 1s ties what Auto-sklearn achieves in 5-60 minutes. 5/9
#AutoML_Conf
update: registration is now open (), along with the call for applications for a small number of travel awards for those in need of financial assistance, . The tentative conference schedule is here:
I'm very excited that my former PhD supervisor
@HolgerHoos
just received an Alexander von Humboldt (AvH) Professorship. This is Germany's highest-valued research price (5M Euro).
The press release focuses quite a bit on on
#AutoML
:
We have multiple positions open at
#AutoML
Freiburg, for both outstanding postdocs and outstanding research engineers. We focus on AutoML methods development, learning to learn,
#NAS
,
#AutoPytorch
,
#Autosklearn
,
@open_ml
, etc. Apply by March 15th; details:
The result of this paper is still astonishing to me: given an RL environment E, you can train a (cheap) neural network “surrogate MDP“ to replace E‘s original MDP to train a strong policy for the original E much faster! Go
#AutoML
&
#MetaLearning
:-)
Paper + Oral accepted to
#AAAI2021
MetaLearn Workshop: We meta-learn proxies for teaching agents Gym tasks, allowing for robust & up to 60% faster training + transferability to unseen agents
Joint work by Thomas Nierhoff,
@FerreiraFabioDE
,
@FrankRHutter
New post in our NAS series: It's on NAS-Bench-101, a tabular NAS benchmark we created with Google to enable reproducibility and scientific experimentation in NAS. We also compared RL and Bayesian Optimization (BO) on NAS for the first time; BO was better!
Last call for submissions to the late-breaking workshop track of
#AutoML_Conf
. The deadline is in 28h (April 20th, AOE):
Submitting to this workshop is a great milestone for AutoML papers in preparation for
#NeurIPS
!
The call for competitions at
#AutoML_Conf2022
is out: We also welcome previous competitions reopening for submissions! Fast-track proposal deadline: Feb 19. Competition start: March 15. Winner announcement: at the conference July 25-27
Are you writing a paper on neural architecture search (NAS)? Reviewing one? See our best practices for scientific research on NAS: , arXiv paper: , blog post:
#automl
#reproducibility
#AutoML
helped the model-based RL approach PETS to find policies that break the Mujoco simulator for half-cheetah with a new rolling gate! We all know that RL is sensitive to its hyperparameters, but this is a new high ;-)
Ever wonder what the limits of current Deep RL algorithms are with better hyperparameter tuning?
The answer (with model-based RL): way better than we thought. It breaks the simulator.
Blog & paper link:
1/⬇️ presented at
@aistats_conf
2021
Congratulations
@jabergT
,
@dyamins
and
@neurobongo
for this runner-up ICML test of time award. This early
#AutoML
work precedes main-stream neural architecture search by at least 4 years!
"Making a Science of Model Search: Hyperparameter Optimization in Hundreds of Dimensions for Vision Architectures" by
@jabergT
,
@dyamins
,
@neurobongo
Efficient
#AutoML
for Off-Policy RL, doing joint architecture and hyperparameter search at almost no overhead by sharing experiences from a population of agents. Similar to PBT, but much faster due to using a shared replay buffer, and also optimizing architectures on the fly.