We will host Machine Learning Summer School 2021, Taipei during the first three weeks of August. We have an outstanding list of lecturers, talking on a variety of topics in modern machine learning. Free registration for students not presenting posters!
A student told me that he believes there might be a connection between the Fenchel conjugate and the Fourier transform. I replied that if such a connection exists, it might be related to the max-plus algebra. It turns out this is indeed the case!
(1/2)
==== My recommendations today ====
Bandit convex optimization
Fast Convergence of Frank-Wolfe algorithms on polytopes
Calibrated Language Models Must Hallucinate
(1/n)
More and more papers I reviewed reproduce existing proofs without proper citations, hiding how incremental they are or “for completeness” when asked about the technical novelty.
I feel several reviewers are not merely rejecting papers but, more precisely speaking, rejecting research directions. To identify them, just check whether their comments apply to all papers in a research direction or not.
==== My recommendations today ====
Isoperimetric inequalities in high-dimensional convex sets
One World Optimization Seminar
Exact Lipschitz Regularization of Convex Optimization Problems
==== My recommendations today ====
Handbook of Convergence Theorems for (Stochastic) Gradient Methods
Sampling via Gradient Flows in the Space of Probability Measures
I did not submit any paper to ICML. But when I read the comments of other reviewers for the papers assigned to me, I feel sympathetic to the authors. The review quality is...
==== My recommendations today ====
An optimal lower bound for smooth convex functions
Inexact subgradient methods for semialgebraic functions
Optimal tradeoffs for estimating Pauli observables
(1/2)
==== My recommendations today ====
Some highlights from the history of probability
Scalable Projection-Free Optimization Methods via MultiRadial Duality Theory
(1/2)
==== My recommendations today ====
Generalization Bounds: Perspectives from Information Theory and PAC-Bayes
What Should we Trust in Trustworthy Machine Learning
(1/2)
Nesterov: It’s very easy to get non- convex problems but for me, this means that we didn’t think enough. The final step, to say this problem is solved—this is where we get convex formulations. Non- convexity is just the first step.
==== My recommendations today ====
Faster Convergence of Stochastic Accelerated Gradient Descent under Interpolation
Sum-of-Squares Relaxations for Information Theory and Variational Inference
(1/n)
==== My recommendations today ====
Majority-of-Three: The Simplest Optimal Learner?
Directional Smoothness and Gradient Methods: Convergence and Adaptivity
(1/n for a large n...)
I have been sharing interesting papers with my students daily for years. Now, I feel it might be beneficial to share my recommendations with more people. The papers chosen reflect my personal research directions and interests, and they are definitely not exhaustive.
==== My recommendations today ====
Optimal First-Order Algorithms as a Function of Inequalities
Gaussian Cooling and Dikin Walks: The Interior-Point Method for Logconcave Sampling
(1/2)
More and more people tend to hide the actual connections to previous works when writing papers. Among them, the relatively honest ones reveal the actual connections in the proofs; the others just reproduce essentially the same proofs without any citation.
TMLR gives the reviewers two weeks to complete their reviews and basically does not allow the reviewers to reject review assignments. Besides being nice to the authors, I think TMLR should also consider being nicer to the reviewers...
==== My recommendations today ====
Large Stepsize Gradient Descent for Logistic Loss: Non-Monotonicity of the Loss Improves Optimization Efficiency
(1/2)
We should find some way to encourage researchers to *not skip* words like "the proof here mimics the proof of Theorem A in Paper B", "below is a direct generalization of Lemma C by D et al.", etc.
See:
- An Explanation for the Logarithmic Connection between Linear and Morphological System Theory
- What Shape Is Your Conjugate? A Survey of Computational Convex Analysis and Its Applications
(2/2)
There are too many new papers in arXiv cs.LG every day... We should start distinguishing between (1) papers on ML and (2) papers applying ML tools to their specific research fields. Not all papers involving math equations are submitted to .
A template for presenting research results:
1. What is the problem?
2. Why is it important?
3. Why is it challenging?
7. What are the existing achievements?
5. Why are they unsatisfactory?
6. What is your result?
7. What are the key ideas in the result?
8. Outlook.
What should I do? Recently we found an arXiv paper that generalized our work, which is great. But that arXiv paper simply cited our work as "a special case," hiding the fact that both the algorithm and analysis they consider are directly adopted from our work. (1/n)
==== My recommendations today ====
Accelerated Objective Gap and Gradient Norm Convergence for Gradient Descent via Long Steps
Optimal Acceleration for Minimax and Fixed-Point Problems is Not Unique
(1/n)
The reviewer errors section of the reviewer tutorial for ICML 2022 is worth reading. I have experienced at least one example per slide in that section...
There are two possible reasons for writing a very long paper: 1) the paper is essentially very complicated; 2) the authors haven't fully understood their result. Nowadays, the reviewers tend to think in 1), so the authors tend to stay in 2).
==== My recommendations today ====
An Elementary Predictor Obtaining 2\sqrt{T} Distance to Calibration
On Averaging and Extrapolation for Gradient Descent
1. Build a review system on arXiv, where experts can share and discuss about their comments about a paper publicly.
2. Reform journals as "recommended lists of arXiv papers" by the editors.
3. Reform conferences as meetings of researchers.
4. Hence no submissions & deadlines.
Let's eliminate the author-reviewer discussion phase and perhaps replace it with an additional round of author rebuttal. It's insane to expect everyone in academia, spread across various time zones, to be available for real-time discussions over several days.
@hanzhao_ml
I once got a similar comment and asked the reviewer to provide a reference. The reviewer did not reply and changed the score from five to one.
I always encourage my students to pursue "normal" (less exciting?) proofs. If we prove a result by an ad hoc proof, we can spend months trying to find a "normaler" one. I believe doing so is beneficial to other researchers, though I know there are so many disincentives...
The meta reviewer says "the proof is standard" and rejects a paper that solves an open problem and empirically works better than existing algorithms. Isn't it better if one solves the open problem with standard techniques rather than an ad hoc, very specialized technique?
Our work provides a novel perspective on online portfolio selection and learning quantum states, improving on the regret analysis of $\widetilde{\text{EG}}$ and proposing state-of-the-art algorithms as applications of *one* regret bound.
Now since there are arXiv and similar preprint archives, we may replace the traditional concept of a journal by regularly announced lists of recommended papers with links. Then, everyone can run their own “journals.” Journals by those with good tastes become the “top” ones.
==== My recommendations today ====
Learnability Gaps of Strategic Classification
An optimal tradeoff between entanglement and copy complexity for state tomography
==== My recommendations today ====
New Brunn--Minkowski and functional inequalities via convexity of entropy
Liran Rotem: Brunn-Minkowski inequalities via concavity of entropy
(1/2)
[Reposting with corrected date!] Math of Data Science "Summer" (Australian Winter) School in Darwin June 24-28, with mini-courses from Gabor Lugosi, Peter Bartlett, myself, Subhro Ghosh, Nikita Zhivotovskiy, and Shahar Mendelson.
Many theory papers claim to have proved what others cannot, but they often fail to spotlight their key ideas. This confuses reviewers and readers, and hints at a lack of novelty.
==== My recommendations today ====
Primal Subgradient Methods with Predefined Step Sizes
Sara van de Geer: A statistician's selection of the work of Michel Talagrand
(1/n)
Is there any name for the fact that the Fenchel conjugate of the relative entropy is a log-partition function? Not sure if Donsker-Varadhan is appropriate...
==== My recommendations today ====
Discussion meeting on ‘Safe Testing’ by Grünwald et al.
Merging sequential e-values via martingales
Improving Adaptive Online Learning Using Refined Discretization
Generalization bounds for mixing processes via delayed online-to-PAC conversions
Improved bounds for calibration via stronger sign preservation games
(5/n)
"However, even after the rebuttal it remained unclear as to why XXX is fruitful, and whether it leads to theoretical or empirical improvement over prior works." After rebuttal it remained also unclear to me whether anyone read the rebuttal...
This is the first time almost all of the reviewers and ACs of the papers I reviewed for NeurIPS are decent. Very surprising.
Unfortunately our submission did not encounter similar reviewers.
It seems that practitioners and theoreticians have different opinions on this matter. Perhaps the true issue lies in enforcing one single system, one uniform rule for all. Maintaining the diversity & openness of the ML community has been always challenging.
Open problem: Direct Sums in Learning Theory
Calibrated Forecasts: The Minimax Proof
Analysis of Langevin Monte Carlo from Poincaré to Log-Sobolev
(2/n)