Meta-scientist
@unibern
. “Jumped-up punk who hasn’t earned his stripes.” All views a product of my learning history. Chief recommender at
@error_reviews
.
It's not just p-values that get hacked.
We added a 3rd large dataset to our preprint: APA PsycTests.
67512 Cronbach's alphas from 60491 scales again show excesses at the .70 rule of thumb threshold.
@maltoesermalte
@rubenarslan
@TaymAlsalti
@frankbosco
Amazing
@zotero
feature I just discovered:
If you highlight text in a pdf, then copy-paste it into another document, it pastes as a quote with a reference and page number!
Working on a meta analysis.
Either researchers are *phenomenally* good at choosing sample sizes to find effects barely p < .05...
Or p-hacking/publication bias have turned the literature to mush.
Can anyone look at this and still think preregistration wouldn't help?
Today is the last day of my postdoc & I don’t have a prof or industry job lined up to sing about.
I don’t see many threads like this, but it must be happening to many given the current academic job market.
I honestly don’t know what I’ll do next, but I’m happy.
Ciao for now.
Today in desk rejections of replications:
Original paper: X is the case. Published in top journal. Hundreds of citations.
Our paper: X is not the case. Failed replication, 7 extension studies showing severity of problem. Desk rejected, told to apply to a specialist journal.
Are citations in the psychology literature accurate? Did the work cited actually support the claim made in the citing work?
Across 3347 citations in 8 leading psych journals, 9.3% of citations were partially inaccurate & 9.5% were entirely false.
BUT:
I just withdrew an accepted symposium from a conference that will now be online
- I would still have to pay to register
- Others would have to pay to view
- Organisation would own the content & restrict access
Why would I agree to this? Esp when preprints & YouTube are free?
Our new paper outlines how neuroscience can be used to:
(1) study the impact of climate change on the brain
(2) identify ways to adapt
(3) study the neural substrates of environmental decisions & outcomes
(4) create better communication & interventions
Our manuscript 'Hidden invalidity among fifteen commonly used measures in social and personality psychology' was just accepted at Advances in Methods and Practices in Psychological Science! w/
@seanhughes102
Many universities require researchers to pay out of pocket and be reimbursed later. Grants are meaningless if you can't afford to pay out of pocket up front - and fragile employment means this is the case for many.
New study confirms that data is NOT available on request.
93% of authors with “data available on request” statements in their manuscript either did not respond or declined to share their data.
New preprint in which we show that there is an excess of Cronbach’s alpha values at common rule of thumb cutoffs (.70) in the psychology and I/O literatures.
@maltoesermalte
@rubenarslan
@TaymAlsalti
How badly can a meta-analysis go wrong? (Very)
How long can the authors ignore serious problems? (5+ years)
What ways can you check an existing meta-analysis for errors? (Let me show you!)
New preprint:
Manuscript just accepted
@Meta_Psy
!
p-hacking - the updating or adjusting of data or analyses in light of prior beliefs about hypotheses - is a cornerstone of psychology research.
I propose a way to streamline it: simply generate random numbers < .049
Kahneman pronounces behavioral priming ‘effectively dead’.
More importantly, he tells us about the effective philosophy of science of our field:
It’s dead because it’s ‘not a wise career move’ not because it’s false.
truth < jobs
Daniel Kahneman, towering figure of social psychology, declares behavioral priming - which he himself was instrumental in making popular - "effectively dead".
What's the largest p value you've ever seen described as marginally significant?
I just found "p = .47" described as “marginal superiority" of one condition over another.
If you’ve ever completed an IAT online, you’ll know it gives you feedback on your level of bias.
@psycholojamie
and I thought it was weird that no one has ever put confidence intervals on IAT scores, so we did it.
It turns out the IAT can say very little about individuals…
I’m so excited to finally get to talk about this publicly!
The Swiss
@snsf_ch
have awarded us $285k to check ~100 important published articles for errors, to foster a culture of error checking & acceptance.
Fewer errors = bonus $ to authors
More errors = bonus $ to reviewers.
Error checking is an important service to science, but it doesn't pay the bills. But what if it did? Introducing ERROR, a bug bounty program for science to systematically detect and report errors in academic publications
A journal has rejected my paper on the grounds that it is 'already published' on a preprint server (
@PsyArXiv
) that that this 'prevents the ability to transfer ownership to the publisher'.
It's 2022. 🙃
Does anyone have resources I can send them?
Overheard in academia today:
Early career researcher: Any advice on how to deal with power dynamics?
Full prof: There are no power issues in academia, and ECRs should feel grateful to have the opportunity to learn from seniors.
🤡
THREAD
I'm psyched to finally announce this. We've curated a very large dataset on implicit & explicit attitudes, beliefs, and identity that we're making available. Journals including
@PsychScience
JPSP,
@CollabraOA
& others have agreed to take
@RegReports
based on it.
I just came across
@MatthewBJane
's Shiny app illustrating the impact of biases and artifacts (reliability attenuation, range restriction, & scale coarseness) on observed effect sizes - very cool learning tool
After you defend for a psychology PhD they take you into a dark room and tell you the truth. "The stuff that’s true is mostly unsurprising and the stuff that’s surprising mostly isn’t true. We just made this all up for grant money."
It’s not just p values that are hacked. The distribution of published AUC values also show distortions at the rules of thumb cut offs.
Nice work
@aidybarnett
and co!
Also applies Cronbach’s alphas:
Our paper on prediction models shows that many researchers are willing to fudge their models to get a "good" AUC. Have you ever been asked to nudge an AUC from just under to just over 0.8? . With
@nicolem_white
,
@RexParsons8
,
@GSCollins
“Power analysis is just something you do to get your manuscript through the editor. You just have to report *a* power analysis, not the one corresponding to your tests.” - coauthor who I don’t see myself working with again
I've changed my mind, Bem was right, we can feel the future.
It's the only explanation for how many authors I have contacted with a data sharing request have "recently wiped their laptop and lost that data".
Data wiping reliably precedes data requests: precognition is real
Very cool new article by
@MarkoSarstedt
et al:
A Many-Analysts style study on how analytic choices in Structural Equation Modelling lead to different conclusions between analysts.
🚨🚨🚨Shameless little bullies alert 🚨🚨🚨
Without my prior permission, a cell of self appointed data police have unfairly scrutinised my seminal work, in an unfair and bullying fashion. They claim to have found an "error".
“one runs the risk of becoming a slave to the concepts that have been generated. Many researchers have taken terms … from everyday language and expect this linguistic categorization to somehow map to identifiable mechanisms in the brain”
I know
#rstats
very well, so why is it so hard to learn
#Python
/
#JuliaLang
on top of it?
- Different IDEs throw me for a loop
- Especially lack of Rstudio’s package help! Where the hell do I get documentation if not here?
- Not being able to run code line by line
Suggestions?
"Crowdsourcing hypothesis tests: Making transparent how design choices shape research results" was just accepted at Psych Bulletin!
This was my first very large authorship study: I am 106th of 193 authors! Cool experience, cool paper.
THREAD:
We are making this journey so that no person will have to make this journey in the future. They will get the proper medical attention they need right here in their own country, free from shame and stigma
#repealthe8th
Remember early career peeps: it’s your mindset that’s the problem, definitely not your material conditions, job security, work hours, etc.
[growth mindset intensifies]
The two most common causes of hardship in PhD students are an inability to accept failure and choosing this career path for the prestige, rather than out of any real interest in research.
Job/Tenure application: I alone can design, run, analysis, and report studies of this brilliance, you must hire me to move science forward
After fraud is detected: I have never even been in the same room as the data, I do not read the manuscripts I coauthor, a rogue RA did it.
A mini Many Analysts:
I taught a masters course on data wrangling and visualisation in R+tidyverse.
Given the same data and instructions for how to do exclusions, this is the distribution of sample sizes after exclusions that different students reported.
Interested in meta-science? Come do a PhD with me and
@maltoesermalte
at
@unibe
in beautiful Switzerland. Open topic although we focus on error detection (see
@error_reviews
) and flexibility in measurement & methods. Email or DM either of us for details. Topic ideas welcome!
Delighted to say I’ll soon be starting a 3-year postdoc working on this exciting meta science project with
@maltoesermalte
,
@rubenarslan
&
@TaymAlsalti
!
(the good kind of meta, not the zucc)
Very grateful to
@dfg_public
to approving the grant proposal "Standardization of Behavior Research Methods" led by
@rubenarslan
and myself as part of the priority program
@meta_rep
. What are our plans for the next three years, besides finding better soccer metaphors? 1/10
Psychological research needs to worry when *video game speed runs* are given more statistical scrutiny than most published articles.
This investigation into a recent Minecraft run contains a good discussion on sampling bias, p-hacking & optional stopping
Speaking only for myself and not my coauthors, this replication project taught me more about the behaviour of researchers and their beliefs than it did about the effect being studied. And what I learned was pretty depressing.
I’m increasingly worried that much of psychology is analogous to Chomsky’s “colourless green ideas sleep furiously”: it has all the syntax of science and you can’t immediately dismiss it out of hand as not being a sentence/science, but it’s meaningless on inspection.
Career trajectories in psychology PhDs (N=1195):
Pipers - start & stay in TT - 17%
Droppers - start & leave TT - 34%
Nevers - never enter TT - 49%
Hoppers - start in industry but later enter TT - 11%
I've seen questions/speculations that many academics are leaving for industry.
I have a paper that follows the career trajectories of ~9k PhDs in STEM who graduated 2000-2008 that looks at exactly this.
Link below, highlights in thread.
Publication bias and hidden moderators: A quick simulation
Imagine a population effect of Cohen's d = 0.2
Simulate 100 studies with Ns of 50 to 150 sampling from this distribution. Meta analysis recovers the population value pretty well: d = .19, no heterogeneity
It's not just p-values that get hacked.
We added a 3rd large dataset to our preprint: APA PsycTests.
67512 Cronbach's alphas from 60491 scales again show excesses at the .70 rule of thumb threshold.
@maltoesermalte
@rubenarslan
@TaymAlsalti
@frankbosco
For any early career researchers wondering if the job market is sane: I’ve published in the journals (psych science, inquiry, and bulletin, JEP:G, AMPPS), won >250k in funding, moved countries.
Just accepted at JCBS:
- power analyses suggest the IRAP literature is severely underpowered.
- much worse than social and personality psych & not improving
- 1% of the IRAP literatures meets minimum power (.80) to detect the *average* effect size in psych (r=.2)
Sample sizes in studies using the Implicit Relational Assessment Procedure are small. How small? Really, really small.
This has negative implications for statistical power and therefore replicability of claims in the literature as a whole.
Behavioral interventions are most usefully designed and implemented by carefully considering the individual’s context, history, and the real-world incentive structures they operate within.
Either it’s a massive coincidence that
@RegReports
produce way more null results, or our traditional publication system serves to pump out huge numbers of false positives
Any researcher who has ever had an article rejected because of null results will appreciate this analysis by
@ChrisPGAllen
&
@neuroccino
now in press at
@PLOSBiology
If you want quality rather than luck to shape your academic career then use
@RegReports
a) this probably doesn’t mean what readers think it does because of a statistical effect called range constraint: for technical jobs, everyone in the room has those skills, there is little variation in them because everyone who answered had to have them to be in that room.
Sample sizes in studies using the Implicit Relational Assessment Procedure are small. How small? Really, really small.
This has negative implications for statistical power and therefore replicability of claims in the literature as a whole.
'Standardized' effect sizes are not nearly as interchangeable as most people think.
In one of my courses I make a similar point, that identical differences in means can produce different Cohen's ds due to range restriction (+ vice versa).
Better population SD norms needed!
Hello friends and foes are you ready for some effect size standardisation discourse well too bad because you're getting it anyway! (unless you scroll away please don't)
Quick thread on a new commentary-like preprint with
@JProtzko
,
@lakens
,
@maltoesermalte
, &
@rubenarslan
Same. I’ve spent 6 years doing open science/methods reform work, and received a huge amount of hostility and criticism for genuine efforts to help.
It has often impacted my mental health, it’s hardened me and made me cynical. I miss being excited about science.
@aeriELLE_allen
I know many professors that would benefit massively from being more willing to admit ignorance, ask questions and change behaviour in light of new knowledge. Keep doing what you’re doing.
Remembering the time when, after presenting results from multiple failed replications, the speaker said they’d keep trying as the original finding must be true “because it was published in Science”
@NAChristakis
@lessig
How could the university have caved to internet outage if it acted months before data colada published their findings online? There’s a basic issue of causality here. Did they feel the future?
To add:
So you're thinking of doing a PhD! Yay! Here's my best advice:
Seriously consider not doing it.
Get a full understanding of the long run economic impact of forgoing good salary and pension contributions for your 20s, and whether it will actually get you a better job.
So you're going to do a PhD! Yay! Here's my best advice:
1) Don't work nights/weekends unless you have a hard deadline. Work 40hrs, no more. My most cherished grad school experiences were hanging out with my friends. Do not waste your youth providing unpaid labor. Don't do it!!
Me: can we get accurate estimates of effect size?
Mom: we have estimates at home, we don't need preregistration, it constricts creativity.
Effect sizes at home:
()
There is a multiverse of possible ways data in a paper could have been transformed, analysed, or reported. Instead of a stale old PDF, what if you could change these options to inspect that multiverse? 2/7
Recently I experienced an entirely novel mode of snark on Twitter that genuinely made me laugh for its inventive pettiness:
Someone added me to a list called “People I will never collaborate with”
I added them to a list called “Likewise”
For anyone who is confident that "Science is self-correcting":
It is surprisingly hard to get a journal to retract *your own paper* even when you are explicit that you are explicit that you have detected data fabrication.
I'm very comfortable working with
#rstats
#ggplot2
ggplot but I must have to google for how to remove a plot's legend *at least* once a week.
Is it just me who has a hole in his brain?
I just got reviews back on a paper. One reviewer signed their review and one did not. Can you guess whether it was the anonymous or named reviewer who was oddly and unnecessarily insulting?
I continue to be pro-signing reviews as it makes us accountable.
“Median statistical power in leading economics journals is very low (only 7%), and excess statistical significance is quite high (19%). Power this low and excess significance this high raise serious doubts about the credibility of economics research.”
I was asked to guest AE for a journal I’ve published in.
Said I was happy to, but that I’d likely ask for power analyses and cautious interpretation of interactions in ANOVAs (which are historically rare in this type of work)
Editor stopped replying a week ago.
“You know, I know this effect doesnt replicate. I know that when I publish it in a journal, the Matrix is telling my brain that it is impactful & gets funding. After nine years, you know what I realize? Ignorance is bliss.
I dont want to remember nothin. Nothing. You understand?”
I am available for hire as an adjunct professor on a without labour basis. Employers must understand that there will no work provided for this position.
@UCAFT_UCLA
u interested?
What would be fascinating is an analysis of another article by the same author(s). To what degree is inaccurate citation an individual differences variable? This may affect how we would try to improve things.
I had to go back and look at the individual scale items for an analysis, and now I'm wondering why anyone should be surprised that they're all associated.
#schmeasurement
My feed is full of people leaving philanthropy for industry. I’m not saying being an astronaut cowboy millionaire is perfect, but I love being one and think it can be a wonderful career option. A thread on why ⬇️
M A T E R I A L C O N D I T I O N S
“Society’s understanding of mental health issues locates the problem inside the person – and ignores the politics of their distress”
I was disappointed to read that "Better Accuracy for Better Science...Through Random Conclusions" is a novel statistical method rather than a call for us to simply make random conclusions, which would be far funnier and more efficient.
This new post from
@DataColada
is quite stunning.
Death threats, researchers who study moral violations engaging in moral violations. My jaw is on the floor.