There is one thing I keep hearing from NIH study sections, in paper reviews, or as questions at seminars: deep learning is not intrepretable and this makes DL problematic for medical imaging. This is flawed and here's something to read about this:
Just received official word that I am promoted to full professor
@Cornell
effective January 1. Iβd like to thank all my mentors, collaborators, and trainees for this amazing journey so far. Onward and upward.
For finding papers, my go-to tool is Google scholar. I try keyword searches, following citations, and author-based searches. I will also add random papers I encounter, say on Twitter, to my stack. 3/
That said, I strongly recommend having a system for processing the literature at a steady pace and not simply reactively. This habit can save a lot headache and improve the quality of your work. Fin.
Each paper added to a stack will be prioritized based on a quick skim of the title and abstract (level 0 understanding). I recommend a 3-stage prioritization. High, medium, low. High, for instance, will indicate a paper that is super relevant to my project. 4/
As an advisor to many grad students over the years, Iβve noticed that there are a few very important personal qualities that determine success. A brief πͺ‘ :
We have gone too far with conferences. They are too big. Reviews are too noisy and not useful. We put too much weight on acceptance. They incentivize half baked flashy papers. They are bad for the environment. And they discriminate against those who cannot travel.
Randomness is so underrated in our culture. Yet it underpins evolution, intelligence, and every single facet of our lives. Some think it's your IQ or effort that determines success. The sobering truth is, pure luck is often the necessary condition that no one talks about.
You can aim to add and categorize ~10 papers a day. That should not take more than 10 minutes. Next, you move on to level 1 processing of your stack, starting from high priority papers. This level of processing should take about 10 minutes per paper and yield β¦ 5/
In our recent paper in
@ScienceAdvances
, we use modern deep learning to show that the fMRI response to naturalistic stimuli is predictable across much of cortex w/ an accuracy close to the noise ceiling. Joint work with
@meenakshik93
,
@ngohgia
@amykooz
A ConvNet paper (with back-prop and applied to a medical imaging problem nonetheless) from 1993(!). This is before LeCun and Bengio. And only 22 citations on scholar. Wow! RT
@affrangi
a basic understanding of the algorithm/model, problem, and data. The inputs and outputs, some modeling details, the literature context (what gap is being filled) will be part of this understanding. You can aim to process ~3 papers/day at this level. 6/
I recommend taking notes for level 1 processing: 3-4 sentences per paper and linking these to pdf and paper stack. Next up is level 2 processing, which can take 1-2 hours per paper. You can aim to do ~2-3 papers a week at this level, dedicating maybe 30 min/day to this task. 7/
Finally we have level 3 processing, where the aim can be replication or baselining. This entails an understanding that allows implementing the algorithm, deriving the theoretical results, and achieving the empirical results. You can probably do 1-2 of these a month⦠9/
You can only decide if a paper should be level 2, after processing it at level 1 and based on how relevant/important it is for your project. Level 2 processing should lead to a good understanding of the theoretical and empirical details, and results. 8/
These recommendations and estimates are all subject and project-specific of course and should only be considered as rough guidelines. Moreover, it does not consider the broad reading that one should also pursue during their PhD and beyond. 12/
Here's what a typical brain looks like as it ages. Warning, this is a little depressing to look at π Consider this a teaser form a manuscript we plan to make public in a few weeks -
@AdrianDalca
I completed the review of a paper. It took me ~5 hours. And a lot of mental effort. In the middle of a pandemic, where every second and ounce of energy is valuable (parents of little kids, you know what I mean). And I did this for free. OA fees or paywalls don't pay for this.
We are gearing up to launching our virtual seminar series in Machine Learning in Medicine - aiming to minimize carbon footprint + speaker burden. Sending out speaker invitations shortly. Will announce line-up in real time. Watch this space and spread the word.
We should educate grad students and post docs about the finances of running an academic research group. Too many, I find, are absolutely clueless about where the funds come from and what they pay for.
We have an exciting faculty job opportunity at the intersection of AI and medical imaging at Cornell, in NYC. Please consider applying and help spread the word.
Itβs good to remind oneself that scientists (even great ones) can be profoundly wrong and we are all prone to our own biases and flaws. Look at this gem from 65 years ago π
Everyone agrees that interpretability or explainability is critical in ML for medical imaging (MLMI). In our brand new paper, we formalize and present the needs and elements of Interpretability in Machine Learning for Medical Imaging:
Progression in ML (1):
*Linear models w/ carefully selected variables can predict
*Linear models w/ lots of variables predict better (but avoid overfitting)
*Kernels can help turn linear into nonlinear
*Nonlinear models do well too
*Nonlinear models on raw data do great
Researchers do virtually all the heavy-lifting in scientific publishing. Largely, for free. And what do we get in the end? Content that readers pay to access or authors pay to publish. Plus, authors often lose the copyright to their own work.
@MELBAJournal
will have none of that.
Missing variables are ubiquitous in real-world data, especially in healthcare. We present Knockout: a theoretically grounded, easy-to-implement and very effective way to handle missingness in machine learning (ML). 1/n
The predictability ceiling is often ignored in mainstream ML research. Every prediction problem has an upper bound for predictionβthe Bayes-optimal performance. If you donβt have a good sense of what it is for your problem, you are in the dark.
We are planning a "green" seminar series in "machine learning in medicine" at Cornell, where the speakers will give their talks virtually. Goal: minimize carbon footprint & burden on speaker. Hope to attract great speakers. Spread word please. And all suggestions are welcome.
We need *high-quality* virtual conferences that are <$50 to register for anyone. If anyone is interested in doing this for ML/Computer Vision/Medical Imaging ping me.
I hate it when journals make you do a ton of work (formatting, following strict word/page limits, filling out forms, suggesting reviewers, editors, etc) just to submit for review - and then desk reject you the next day.
1/ stick-to-iteveness: research is full of road blocks and can at times be unpleasant. Getting concrete results often demands persistence, creativity in problem solving, clarity in your vision and conviction.
A typical applied ML paper these days: we ran experiments on 10 different datasets, using 20 different architectures, each with 100B learnable parameters, using 1000 GPUs for 20 weeks straight and found that a mini batch size of 8 works fine most of the time.
I know academics tend to complain about teaching and grant writing. But, I canβt imagine better incentives for organizing oneβs thoughts and learning new material.
From Connectomic to Task-evoked Fingerprints: Individualized Prediction of Task Contrasts from Resting-state Functional Connectivity. Our paper w/
@ngohgia
@AmyKuceyeski
@meenakshik93
:
First three stages of a research project: 1) irrational exuberance (no one has thought of/observed this before), 2) denial (oh that paper is close but not the same), 3) acceptance (how can I pivot).
The different goals ppl have in mind when they say "to interpret a model". 1)What can I change to alter outcome?, 2)what part of this data-point influenced this decision (most)?, 3)which historical (training) data-points influenced this decision?, 4)show me typical failure modes
We should be discouraging the rat race mentality, not encouraging it. We should be incentivizing deep understanding, not superficial creativity. We should be emphasizing long-term planning, not short-term results.
Are you a PhD researcher in the field of computational medical imaging? Are you looking for faculty jobs? We, at
@WCMRadiology
, are hiring. Please read on. π§΅1/
Thanks everyone for the kind words. I wanted to say two more things. First, I canβt be happier to call
@cornell
my home. This is an amazing place and Iβm very excited about building up Engineering in NYC. Second, I have many role models that inspire me and hope to thank some day.
Just received official word that I am promoted to full professor
@Cornell
effective January 1. Iβd like to thank all my mentors, collaborators, and trainees for this amazing journey so far. Onward and upward.
I watched the GWAS community obsess over statistical power and grow their sample sizes to the millions, using univariate tests to detect associations between single genetic markers and complex phenotypes. We now have hits that βexplainβ a tiny fraction of the variance.
We are planning our UnJournal movement, which can help free us from the constraints of conventional scientific publishing models. I dream of the day when "UnJournal" catches on as a word and represents a new way of scientific dissemination :) Let's see....
2/ good communication: you canβt have an impact unless you convince other people to be inspired by your idea and build on it. Your work is as valuable as you can communicate it - in writing and verbally.
This is shameful. Scientific publishing companies that have historically enjoyed outlandish profit margins, largely on the back of the hard work of academic researchers who are often supported by tax payer money, should not be able to get away with this.
In this scenario, the cost of publication is covered by an Article Processing Charge (APC) paid at the time of publication. The APC for Nature Neuroscience in 2022 is β¬9,500/US $11,390/Β£8,290.
Nips sold out in less than 40 min. I repeat, an academic conference is sold out at about the same rate as a Radiohead concert. If this ainβt an indication of an ML bubble, what is?
Please stop using convolutional layers for problems with no shift-invariance/equi-variance. If (1) you do not understand what this means, (2) you're at Cornell and can take classes - consider taking ECE 4250 (DSP) in the spring.
Many top-tier scientific journals are published by companies that care about revenue. Thatβs why we have the obscene dichotomy: scientific articles behind unethical paywalls or outrageous ($1000s) article processing fees charged to authors. There is another wayβ¦
Iβll make it very simple: any high profile applied (eg medical) machine learning paper should evaluate the tech in a real world scenario, prospectively. If thatβs not the case, it doesnβt deserve public and widespread attention. Period. Journalists, get that?
I will be recruiting multiple PhD students from the ECE, CS, and BME programs in
@CornellEng
. Application deadlines are soon! If youβre interested in the intersection of machine learning, computer vision and biomedical imaging, consider mentioning that youβd like to work with me.
I'm looking for a post-doc w/ a strong machine learning and computer vision background who'd be interested in working on a prostate cancer imaging problem. This would be in New York City, on the
@WeillCornell
and
@cornell_tech
campuses. Please DM with leads. Email me w/ CVs.
If you know anyone who has a track record of working at the intersection of neuroimaging, machine learning, genetics, and Alzheimerβs, and they are looking for a postdoc, can you hook me up? Thx!!
Today will be my last lecture to
@CornellEng
undergraduates, as I am moving from Ithaca in a few weeks. You can find me hanging out on the
@cornell_tech
campus in NYC soon.
Reference letter from non-US Prof: "This person took my class and received a respectable B-. We wrote a semi-decent paper together on this topic. I think they are qualified for the position you are considering them for."
When, in grad school, I first started reading scientific papers, my aim was to absorb/understand. It took me years to change my mindset and have a critical perspective. Now, I try to teach my students that one should question everything in a paper. Even best papers are flawed.
This is a paper we worked on for almost 3 years! Machine learning can aid in prediction of IDH mutation from H&E-stained histology slides in infiltrating gliomas | Scientific Reports
Many training datasets have noisy labels, especially in biomedical applications. Zhilu Zhang and I propose an easy-to-implement loss function that can make your classification model more robust to noise. The paper is under review.
4/ flexibility: as any good list, this one introduces a contradiction - in particular with (1). While persistence is important, sometimes you are at a dead end, and recognizing that and changing course is critical.
IMO, this way of thinking can deplete resources, kill scientific creativity, and reveal very limited insights with little practical value. I urge the neuroimaging community not to fall into this trap.
Open access is a must in scientific publishing. Another must is high quality, well curated content. Finally, there is no need for high publication fees, which amplify inequalities.
@MELBAJournal
will be built on these principles.
For 3+ years now, since I moved to Cornell, I have been holding a ~weekly journal club. Many of the papers we read shape our views and research. If you are interested in taking a glimpse of what we have read or are reading, here you go: