Biostatistician at Memorial Sloan Kettering Cancer Center. Special interest in prostate cancer, risk prediction, patient-reported outcomes, decision-making.
The Charlson Comorbidity Index is based on a single study of 559 patients at one NYC hospital in 1987. That’s why HIV is six times worse than diabetes. Two questions: 1) why do scientists not check their sources? 2) why are we still using this?
Statistics education should abandon teaching of computation (eg calculate z from the data) and focus exclusively on conceptual issues (eg inference vs. estimation; what are multivariable models?) and how to interpret statistical results (eg p=.063, conclude what?). Discuss.
I am studying informed consent for clinical trials, wanting to see how we can simplify this process for patients. We give a brief questionnaire asking about their consent experience. But I have to first give patients a consent form that is 10 pages long.
This is the normal Ioannidis thing about research being terrible. The article has a fatal flaw though: he says we shouldn't take a certain action without good evidence, but NOT taking that action is also a decision that requires evidence.
A fiasco in the making? As the coronavirus pandemic takes hold, we are making decisions without reliable data
Op-ed in
@statnews
by John Ioannidis
@METRICStanford
I generally dislike papers whining how about bad published research is, but this is important. Machine learning in medicine seems to exist well outside of the mainstream of established methods: ~90% internal validation only, ~95% ignore calibration.
Generic statistician type of moan: Please stop using Bonferroni correction when testing a limited number of correlated hypotheses (eg is there an effect on anxiety? what about depression?). I know it makes y'all feel very statistically rigorous but it is unhelpful and invalid.
I asked the editor of a medical journal out on a date. She said no, but told me I could transfer the date request to her sister, who would charge $3,500 if she accepted.
Big news! has been completely updated and revamped. Code, tutorials, guides, bibliographies, videos and more! Amazing job by
@ShaunPorwal
@statistishdan
Prediction modelers: please be like
@ashclift
@JuliaHCox
@GSCollins
& write papers like . Large data set, careful evaluation of different modeling approaches, evaluation of calibration (which appears not to exist in ML world), decision curve analysis
Proposal: clinical journals should replace the "conclusion" section of abstracts with "Implications for Clinical Practice" and "Implications for Research". One key point: Perfectly fine to have "implications for Clinical Practice" be "None".
Surgeon: Surgery better than radiation? Great paper! Radiation oncologist: Radiation worse than surgery? Paper is trash! Methodologist: the study was well done, though more exploration of residual confounding would be of benefit.
COVID has been taken as evidence of so many things being broken, like our healthcare system, our government and our supply chains. But what about clinical trials? Over 3m cases but only a trivial number on trials. Clinical trials need a top-to-bottom rethink.
Just sat through another 45 minute presentation on a Big Data / Machine Learning project, where goal is about clinical decision-making and YET AGAIN, endless discussion of data flow, database architecture, boosting algorithms etc and zero reference to bias.
Avoid ending your paper with a vague call for “further research”. You should be able to specify next research steps based on your findings (otherwise, what value was your paper?). Also remember that further research on one subject means not doing research on another.
Can we declare war on acronyms? I'm tired of reading stuff like: "[in] men with csPCa in T-PBx, csPCa was also present in NT-PBx [in 17.2%]". Rule of thumb: if you wouldn't use an acronym in speech, don't do so in writing.
@KristinSainani
I was taught that Florence Nightingale was the first nurse, and went around mopping the brows of fevered soldiers. She was, in fact, a statistician and based her recommendations largely on data. …
Keep getting told "I use MRI to follow active surveillance pts. If MRI is clean, I forgo the biopsy." Never was an evidence-based strategy. We now show it is harmful.
Quality of code in biostatistics is poor. . Key points: ~1/3rd papers with complex stats didn't use statistical code, 0 of 16 sets of code scored even "moderate" on 3 basic criteria for good coding practice.
I’ve heard it a million times: don’t give a doctor a risk prediction for a patient, they can’t handle it. You have to give them a cut-point and put patients into hi vs. lo risk-groups so they can make an easy clinical decision. Any reason to believe this argument is true?
We graduate Masters’ and PhD statisticians with great math theory, but little biomedical knowledge, rudimentary coding skills and often poor language proficiency. We reward first author papers about esoteric methods but not second author papers that advance biomedicine 2/3
How heterogeneity & risk of bias is typically done in meta-analysis.
1. Calculate I2, Cohran's Q, p value for each meta-analysis
2. Report a risk of bias table
3. Completely ignore the findings of 1 & 2 in results, discussion & conclusion
@Richard_D_Riley
Common practice & recommended in several papers is to biopsy man with -ve MRI if PSA density > 0.15. But where did this cut-point come from (brief answer: a pretty random paper in the pre-MRI era) and is it rational (brief answer: no)?
The discussion by biostatisticians about poor stats understanding of investigators is directed outwards. We need a hard look in the mirror. We have been prioritizing abstract math over scientific collaboration for years, in graduate education and in our promotion structure 1/3
Question from clinician: "Andrew: can you look at this study and explain the statistical analysis to me? We want to write a letter to the editor complaining that the paper is unsound and want to know what to say".
All statisticians & data scientists training for healthcare research would benefit from dedicated time observing clinicians, GPs & health professionals make decisions & seeing the pathway involved
Should be part of any MSc in med stats/health data science, & training fellowships
Douglas Altman (12 Jul 1948-3 June 2018) UK. “One of the most influential medical statisticians of the past 50 years”. Bradford Hill Medal 1997, BMJ Lifetime Achievement Award 2015, for spearheading reform of medical research reliability & reporting. 1/2
Nice summary of the PREVENT randomized trial showing no significant difference in infections, retention or bleeding between transrectal prostate biopsy with targeted prophylaxis vs transperineal biopsy without antibiotics -
@EdwardSchaeffer
#DSUIOnTheBeach2024
A discussion I’m having right now: can a university in Sweden transfer completely de-identified cancer data to a US academic institution and keep a key in case of data questions? No, because of a theoretical privacy risk. Insane privacy regulations are killing cancer research.
General advice to the academic community: make a choice between easy fame (dramatic findings!) vs doing hard work carefully and cautiously. Former is a nice dopamine hit and great for your career (though it shouldn't be). The latter is a pleasure you get to live with every day.
I have been corresponding with the authors of the well-known Santa Clara County COVID-19 preprint, and I am alarmed at their sloppy behavior. The confidence interval calculation in their preprint made demonstrable math errors - *not* just questionable methodological choices.
The data, including from randomized trials, are very clear that DRE is not of value as a screening test (though of use in work up of a man with elevated PSA). We can’t have clinical observations take priority over data.
I continue to diagnose prostate cancer in men with normal PSA based on prostate exam alone. Please do prostate exams on men 55-70 who present for a physical. It's not a violation. It's a life saving, inexpensive test.
Can I say this any more clearly? The idea that >3 positive prostate biopsy cores or >50% in one core portends a dramatically worse prognosis was pulled out of the ether and is not based on empirical data.
Really, really baffled by the argument that we should call Gleason 6 "cancer" because biopsy might have missed Gleason 7. Is there any equivalent in medicine where we call a non-finding a finding just in case we missed the finding?
When evaluating a prediction model: discrimination and calibration are for the analyst, to help improve a model; clinical utility (decision curve or other) is for the clinician, to help them decide whether to use the model.
So let’s get this right: send 1400 letters, do 300 genetic tests, biopsy 18 men and find 7 low risk cancers. A lot of work and cost with zero benefit and harm 7 men. How is this a success?
In England, prostate cancer incidence strongly predictable by regional wealth R2=0.71 (data from , excludes London). NHS "informed choice" program on PSA is actually a "screen the wealthy" program.
Here we go again. Urology finds a new way of finding cancer (see PSA, MRI) then ignores “relevance of a finding depends on how you found it”. PSMA finds high prevalence of skeletal mets, even in men with low risk cancer. Likely huge overdiagnosis.
We cannot research and address racial disparities in prostate cancer unless we understand the mechanisms behind racial differences in incidence and mortality. Published today in the Journal of Clinical Oncology
@BrandonMahal
@seunogunwobi
@f2harrell
. The news, by dichotomizers. Fewer than 100 die in a crash on I-95. The unemployment rate is above 10%. The president’s approval rating is less than 50%. In tonight’s game, both the Yankees & the Red Sox scored more than 4 runs. Tomorrow's temperature will be above 65.
At
#EAU23
and pretty stunned at the near complete failure of pretty much every speaker to present data, data and figures following what are, after all, the European Urology guidelines
Trial shows effect size d, p=0.04. True effect size is indeed d. What is probability that exact replication (design, sample size, treatments) will have p<0.05? Actually ~30 – 50%, a result with interesting implications for the replication crisis.
Metastatic prostate cancer lesions found by PSMA PET alone less aggressive than those found by bone scan. Why do we keep forgetting – PSA, MRI – that the aggressiveness of prostate tumors is not independent of how we find them?
Amazing how those who harp on about freedom have so little understanding of what freedom actually means. This couldn’t be any closer to “the freedom of your fist ends at the tip of my nose”. This nurse is in fact free not to get vaxxed. Just not free to endanger patients.
I questioned the results of a high profile paper. Statistician sent me their code. It failed to follow any of the most basic principles of good coding practice. We’ve published on this: if you write code (and all statisticians do) do it properly!
Pattern 3 prostate disease cannot metastasize to nodes (Ross et al 2012), or seminal vesicles. But, rarely, it penetrates the capsule. New evidence suggests that this is not biologically relevant: EPE in Gleaosn 6 doesn't raise BCR risk
@uroegg
Quick! Easy! How to be very scientific when you report your observational study! Just add the following to the Discussion section: "Correlation does not imply causation", "unmeasured confounding cannot be completely ruled out" and "further studies are warranted"
I once peer reviewed grants for a Canadian fund where PI got on a call with peer reviewers. I realized that I’d misunderstood a few things, but also got to explain to PI why I thought certain changes were needed. All very collaborative and civil. Why can’t NIH be like that?
Spent >1 year trying to share data with European co-PI of NIH grant. Data sets fully anonymous, zero patient risk. Just told regulatory burden so high we should attempt novel informatics approaches to run analytic code remotely. Overzealous privacy regulation hurts patients.
If you are submitting a paper to European Urology
@EUplatinum
(or any other urology journal for that matter) please follow the statistical guidelines. I really shouldn't have to start every statistical review saying "please follow the guidelines"!!
@f2harrell
@mike__katta
since year dot: "we should tell clinicians probability of disease." Response: "nah, too difficult, just give a yes / no". New study showing that "yes / no" leads to bad estimates and hence harmful medical practice.
Did stats for observational study in informatics. Authors included thoughtful discussion of causality, concluding likely causal effect. Copy editor trashed it citing JAMA policy “causal language ... only for randomized trials”. Surely peer review/scientific not editorial issue?
Before you make a knee-jerk response "correlation doesn't imply causation: avoid causal conclusions from observational studies", remember this is what tobacco companies said for years. Be thoughtful about causal inference, don't abandon it
@_MiguelHernan
NEJM has not one, but two studies on colorectal cancer screening that report only sensitivity and specificity of a new test. I have zero idea how to interpret the results: are they good tests or bad?
#ClinicallyRelevantStatistics
The rationale for removing the cancer designation from Gleason pattern 3 is 100% practical. What will happen if we continue to call pattern 3 cancer & what will happen if, instead, we use a label other than cancer?
@uroegg
@dr_coops
Very nice to see. Message to AI / ML folks: "Evaluating ... a model [means] estimating model discrimination (eg, c statistic), model calibration (eg, calibration plot, calibration slope), and clinical utility (eg, decision curve analysis)."
Anyone else tired of seeing cost effectiveness analyses / Markov models where the inputs are based on huge amounts of extrapolation / guesswork ("we assume that...") and then the authors reports QALY's to three decimal places (i.e. 8 hours) and costs to the nearest dollar?
Just saw *another* paper in which a sophisticated biomarker was dichotomized, you know, just because that is what we do right? So thought I'd tweet this paper which is pretty widely cited.
Why are we still reporting data in journals pretty much as Bradford Hill and Fisher did it? Why not take advantage of the interactive properties of web publishing?
The problem with MRI-targeted biopsy is that grading and treatment guidelines stayed the same while the diagnostic pathway changed. More evidence that we need to update guidelines (or at least, follow the updated guidelines).
This is interesting, and perhaps pretty transformational. Evidence that what matters in prostate cancer is total amount of pattern 4. Ratio pattern 4 to pattern 3 (how we currently grade) not so helpful.
Lemma: the amount of work involved in conducting a statistical analysis of some investigator’s data set is independent of the scientific value of that analysis.
Some thoughts on Cohen’s d (standardized differences): if my boss gives me a $1000 pay rise, whether I should be pleased or not depends on the standard deviation of salaries at my hospital, right?
Can I 100% clarify for the record: I don't advocate against MRI-targeted biopsies. My view is that MRI targeting PLUS current guidelines on pathology grading PLUS current guidelines on treatment causes overtreatment.
On academic credit: the greatest satisfaction in an academic career is *not* getting credit for one of your ideas. It means your idea has been absorbed into the body of scientific knowledge.
Also — I read Andrew Vickers' “What is a p-value anyway? (In just 2 days)
Review: It is INCREDIBLE. Format is 3 pages per chapter than 1-3 questions w/ answers. Not much math. Great stories
Highest recommendation
#BeImmuneToBeingBamboozled
@KeithKow
I disagree with the letter. I don’t agree that “it shouldn’t have been published”. Publication makes clear in black and white that these attitudes are found amongst the leaders of our field.
To make it 100% clear: data showing MRI-targeted prostate biopsy leads to overdiagnosis & overtreatment doesn’t mean MRI-imaging not useful for eg. avoiding biopsy. Likewise, pointing out the dangers of apricot pits is not to denigrate the health value of fruit.
Can we please stop citing p values of 2.55 x 10^-191? This is the probability of randomly selecting the correct atom in the universe, if every atom in the universe had another universe inside it.
@JamesSurowiecki
I’m sorry, but this is wrong. They did say “don’t go out and buy them” because they wanted to preserve them for medical professionals. They never said they don’t work. They said they do work on April 2
0.15 ng / ml / cc cut-off for PSA density came from the pre-MRI era and has never been justified using empirical data. We show that it is likely not the optimal threshold for biopsy in men with negative MRI.
Prostate-specific Antigen Density Cutoff of 0.15 ng/ml/cc to Propose Prostate Biopsies to Patients with Negative MRI: Efficient Threshold or Legacy of the Past
@f2harrell
I usually make the argument that observational research was not discovered at Google in 2005, that epidemiologists and biostatisticians have been doing this stuff for quite a while and that, you know, there might be some lessons there.
#StatsTwitter
anyone noticed a huge double standard with respect to stats vs lab methods? Stats methods said to be "confusing" with request to "do something simpler" on papers where complex lab methods are inscrutable to anyone but a hard core expert in that narrow lab field?
Many papers recommend "shared decision-making" or "discuss with patient" where findings are not 100% unambiguous. I'm increasingly seeing this as a way to shirk responsibility because shared decision making etc. generally has not been evaluated in the scenario at hand.
Pre-prints were developed as a way of doing better science - get some feedback, inspire new ideas, stimulate replication studies - before formal publication. Now they just seem to be a short-cut for media hungry scientists. So, please,
@Richard_D_Riley
, do stay old-fashioned.
Maybe I’m old fashioned, but I don’t think it’s appropriate to be releasing press releases when submitting a pre-print. Dangerous even. Surely a paper needs to go through peer review first? Once a message is in the media and public domain, it’s very hard to turn back.
Brilliant starting disclaimer from John Ioannidis
#WCRI2024
keynote that as someone who works on bias, he may have a lot of biases. Channeling his “why most published research is false” paper, he says his work should be scrutinized the same way