"Methods and Algorithms for Correlation Analysis in R".
I admit I did not know there are all these variants of correlations. 🤯 Multi-level correlations?
Part of the easystats R package. Check it out:
Preprint -
Github -
A reminder for the new academic year:
We have a collaborative "Effect sizes and confidence intervals guide" to ease your/students' lives regarding all effect sizes related (examples in R):
Use/adjust/adopt/contribute as you like.
=Effect Size, Confidence Intervals, & Power Analyses Collaborative Guide=
@qinyu_xiao
did remarkable job revamping our collaborative guide.
Extends
@lakens
(2013) ES guide to updated methods & using R.
Please use, check, & give us feedback!
We social psychologists really need to discuss "Humankind" by
@rcbregman
.
I finished listening to audiobook yesterday. Putting aside inspirational message, I think social psychologists should read chapters regarding the classic findings in social psychology.
Explanation👇
🧑💻Beginner R/Rmarkdown workshop completed 🧑💻
Simple powertools making R easy with copy-paste. Get you started, overcome scary transition to R.
No coding required to do amazing things, get started with point & click, build skills, learn, improve.
Video:
= Moving the needle with replications =
Submitted replication of Psychological Science classic to PS. Found errors in reporting in original, reproduced very brief methods in original, various extensions (between/within designs).
Desk rejection: haven't moved the needle 🤷♂️
=grant reviewers as gatekeepers hindering replication work=
Two years of Hong Kong's GRC grant requests for doing replications+extensions.
Last year: Score 3.5/5
Asked clarifications, I resubmitted with details.
This year: Score 2/5. Fascinating feedback.
Examples below 👇
Have you watched "The Prevalence of Replications in Psychology"?
Beth Clarke shares the result of their analysis.
What's our replication ratio? It's 0.2%. No, not 20%, not 2%, it's 1/5 of a percent. 0.2%. 1 in 500 gets replicated.
Video:
A reminder for the new academic year:
We have a collaborative "methods best practices" where we keep track of methods resources (~150 pages):
Use/adjust/adopt/contribute.
(If you contribute, add your name & contribution and become a coauthor)
This came up on my YouTube, and I ended up listening to all of it:
"Research methods lecture on internal, external, and statistical validity" by Simine Vazire.
Recommended.
I learned a few things, and found new tools and slides for my teaching.
= Old school journals and Registered Reports =
Case study thread 🧵
We had a Stage 2 Peer Community in =Registered Reports= endorsed.
Was very surprised when JEP:G editor, which is an "interested" journal reached out and asked us to submit to them.
We did.
👇
We ran 4 replications of studies originally published at Journal of Risk & Uncertainty (JRU), & submitted to JRU.
Decision: Desk reject
Reason: "Sorry to tell you that Board decided not to pursue paper further. They believe it is better suited to psychology journal."
🤦♂️🙅♂️
Sigh
😯
Never happened before.
As part of a review for PCIRR Stage 1 a reviewer suggested randomizing direction of scale, we opted to stick with original's, & in round 2 she ran part of our experiment randomizing order to test possible impact (!).
That's next level reviewing. 👏👏
To me, grants are such an absurd wasteful academic cycle. Example I noticed below. Such as waste.
Conclusion here was "Keep submitting".
I think we need to rethink it all.
Have other examples about the wastefulness in grants process/requirements?
Please share.
Junior scientist struggling w/ grants asked for advice. I agreed to help them out, but also wanted them to see this image.
This image shows the grants that I’ve been involved with going back to 2012. The green dots are the successes, the red dots are failures.
Keep submitting.
= open teaching materials for your courses =
Few have reached out to me recently to ask about how I run classes at HKU.
Here is a thread of all resources we have to help you in your classes & consider mass mobilizing students for open & meta science.
Thread 👇
Unbelievable. Prestige and reputation cost us 2 billion US$ a year. The things we could have done and achieved with this money.
I can't believe we just accept this and go along with this as if this is normal.
This is not normal.
😔
A new preprint estimates "that, globally, a total of $8.349 billion ($8.968 billion in 2023 US dollars) were spent on [article processing charges] APCs between 2019 and 2023.
Trying to make a list of open-science supportive scholars working in the field of judgment and decision making, social cognition, and/or behavioral economics.
Know any? please tag them below. Are you one? please reply below.
(I'll share the full list with those interested)
= Original authors responding to replication attempts =
Shafir & Cheek respond to our (& Many Lab 2's) failed replication(s) of Shafir (1993).
We are writing a reply, but it's worthwhile to share some highlights from this exchange.
🧵👇
@adfillon
That you took the time to look into our article, look through the code, find the mistake, and report it to us, is what real science looks like. Exceptional.
This often goes unrewarded/unacknowledged. Wish I could do more to support this work.
Thank you 🙏, keep going strong!
Gave 3-hour workshop on JAMOVI/JASP/R at
@hku_science
yesterday. I thought maybe some of my students would come. 150-200 from all faculties showed up.
Cloud Drive for workshop with PPT & lots of goodies:
Our collaborative guide:
A reminder for the new academic year:
We have a Registered Report template to ease your/students' life writing manuscripts, especially (but not only) for replications:
Our CORE team (with many UG/MSc students) used this successfully in many submissions.
=Replications & angry original authors thread=
Yesterday, we received decision letter on our replication & extension journal submission. It was revise & resubmit.
Good news, right?
Afraid not.
Signed Reviewer
#1
, original authors, went on a full-on rampage.
On that below 👇
Just found out that the psychology job market got a boost from someone finally setting up a Google Sheet:
Don't know who did this, but whoever you are, THANK YOU!
I really hope this will pick up, to be as good as the management sheet.
Help improve this!
= Psychology job market upgrade needed =
"Psychology Job Wiki" appreciated, but you have to see what's happening in OB/Management market.
We have to share FAR more information to make psych market better.
Learn from this:
Requires community effort.
We get many rejections from various journals asking us replicators to fix original work they published, resolve mixed/failed findings (outcome based decisions) & add 'contributions'/'value'.
Decade has passed, still very few replications published. Unsustainable, has to change.
Know meta-analyses?
Help improve "Resources for Learning (/Teaching) How to Conduct Meta-Analyses" GDoc. It's coming along nicely. I moved many of my syllabi there, & I add things when something comes up (e.g. today - mixmeta &
@dsquintana
meta videos)
=Effect Size, Confidence Intervals, & Power Analyses Collaborative Guide=
@qinyu_xiao
did remarkable job revamping our collaborative guide.
Extends
@lakens
(2013) ES guide to updated methods & using R.
Please use, check, & give us feedback!
Finally some good news for open-science at HKU.
"HKU Registered Reports challenge" was approved, aiming for motivating cultural change at HKU to promote
@RegReports
& working with students, funding students' in-principle accepted RRs up to HKD$8000 per student, 30+ overall.
👇
Gave "JAMOVI, R, & JASP" 3 hours
#OpenScience
workshop yesterday.
Covered:
-
@rstudio
is easier than u think, teaser demo power of R
-
@jamovistats
demo: data wrangling, experimental, correlational, meta designs
-
@JASPStats
briefly
12 JDM replications: Results are in!
Successful: 9/12 (75%)
1 Mixed + 2 Failed (with good reasons/insights)
Convergence, comparing results of...
2 teams per each replication: 11 of 12
MTurk US vs. Prolific UK samples: 8 of 8
More info in thread below 👇
= Evaluating scientific evidence guide =
Started yet another collaborative guide since couldn't find a guide to train my students how to evaluate scientific evidence.
My initial attempt is outlined in:
Help is welcome and much appreciated. 🙏🙏
Results are in, very interesting!
Thanks for voting.
I'll start with the strongest effect here.
Effectiveness/efficiency wise:
I do not understand this one at all, but it's SO strong.
2 candidates, same outcomes:
A spent millions, B got it done with no budget.
Why is A>B?!?!
Two candidates, with same number of publications (let's say, 10) from the same journals and very similar qualifications.
A: Secured and led 5 external grants
B: No external grants
Which of those SHOULD your department hire? Why?
🔬=Reproducible Meta-analysis Registered Report template=🔬
1. We face serious issues about reproducibility of meta-analyses.
2. We need to move towards model of meta-analysis Registered Reports.
=Join us in creating meta-analysis open-science RR template!=
Details below👇
Bit hesitant to share this, but thought it's helpful to layout rejection path leading to JESP publication for recent project.
Most by editors strong supporting of open-science.
Referring to this article:
See rejection thread below:
Inpress at JESP (w/ Jieying Chen)
"Regret-action effect: Action-inaction asymmetries in inferences drawn from perceived regret"
4 experiments (3 prereg) w/ consistent effects
"regret perceived more likely result of taking action than not acting"
Link:
Dec 20 I will summarize findings from 18 replications we conducted this semester. We wanted prediction markets, but were overwhelmed by work just running the replications.
Instead, a Twitter vote, you're welcome to rate replicability.
Below are studies we attempted to replicate.
Please write APS at editorsearch
@psychologicalscience
.org and nominate Simine Vazire and Chris Chambers for Chief Editor of Psychological Science.
We should try and support open-science supportive editors for such roles.
Maybe, just maybe, if enough people write in...
APS invites nominations for Editor of
#PsychologicalScience
, one of the most influential empirical journals in the field. Submit nominations by Friday, June 2, 2023.
Ever wondered what John Bargh thinks about the replication crisis & outcomes of replications of priming research?
Then you might be interested in
@behavioralgroov
interview with Bargh:
I find this absolutely fascinating. 😟🤔
Evaluations are truly humbling experience & window into human psyche.
Teaching is hard, especially on crisis.
Sure, evaluations tricky, bad>good, can't please all, & all that. Still, I feel for these students, understand their frustrations.
Hope to do better next time around.
= statistics best practices guide help request =
For upcoming semester, trying to start collaborative guide for best practices in statistical methods. I couldn't find anything suitable, know any?
See/join/edit:
What resources do you know I can add?
SPPS not accepting meta-analysis submissions?!
@siminevazire
:O
Reject decision letter "I do not think that meta-analysis is a valid technique".
This is generally worth discussing. And, if this is policy, it would save everyone time to put it on author submission guidelines, no?
= Replication publications in reputable psychology journals =
Let's talk about replication in psychology. We think we've been doing better in the last decade, but have we improved?
Thread below. 👇
🚨=List of free collaborative resources=🚨
We developed a lot of free collaborative resources that you can use in your research/teaching that you might not be aware of.
Use/contribute/share as you like.
Full updated list:
See list below 🧵👇
This is our... 4th rejection of replications from Psychological Science, all had nothing to do with our rigor, but about "contribution".
We afterwards found homes in other "specialty journals" but obviously failing at attracting the attention originals are still getting. 🤷♂️
= check me, replicate me, I'll help & pay you =
Seeking help in validating & replicating my work. It's time to publicly invite 'red-team' approach & replicators to help me do better in my research.
First draft:
Feedback? Can I improve this call?
= Psychology job market upgrade needed =
"Psychology Job Wiki" appreciated, but you have to see what's happening in OB/Management market.
We have to share FAR more information to make psych market better.
Learn from this:
Requires community effort.
Just made aware of PSPP, the open-source completely free alternative/clone to SPSS.
Do try & ditch SPSS, yet if somehow lecturers/department still insist on SPSS atleast offer students PSPP free open alternative.
Tried this on an old project, works.🤷♂️
Very surprised that most of the reactions to this are that this shows we need to blind authors & even less openness.
I disagree. Strongly. Very strongly.
We must open up the peer review system. Open and identified are important for accountability & learning.
Reasons below 🧵👇
I want a very simple story to be true: More transparency is better for science, always.
I rebut most arguments against the simple story with evidence, theory, or really good rationalizing.
Status bias in peer review is the most challenging reality against what I want to be true
My personal conclusions:
JEP:G does not understand
@RegReports
.
JEP:G does not understand
@PCI_RegReports
.
Editorials are one thing, actions are another.
I hope our experience would benefit others next time around.
My suggestions to PCIRR & for you if you get invited:
How does
@amazonmturk
(w/
@CloudResearch_1
) do compared to
@Prolific
?
Please look at effects in the same replication we ran with Brits in Prolific (0.57 [0.49, 0.64]) & Americans on MTurk (0.59 [0.51, 0.66]).
Near identical!
Atleast for JDM, MTurk almost same as Prolific.
Accepted at
@CollabraOA
:
"Self-interest is overestimated: 2 successful pre-registered reps & exts of Miller & Ratner (1998)"
Congrats
@CameronBrick
@adfillon
@yskjdm
& 4 students & well done!
PP:
OSF:
Mindblowing.
Showed this to students, they were 🤯🤯😲😲.
If you haven't tried it, take 5 minutes, have a look.
It's free, and we, in Hong Kong, are GPT/AI blocked, so this solves 2 major issue for students in HK wanting to leverage AI for super easy data analysis.
Brilliant.
Talk to your data via AI using
Massive upgrade in v0.98:
GPT-4 enables better code,
Explains codes, results, or statistics in general,
Upload a 2nd file,
EDA reports!
Demo:
Just noticed this inpress at Collabra on Researchgate:
"A Multi-Site Collaborative Study of the Hostile Priming Effect". Probably missed announcement somewhere.
Wow (for effort!), ouch, & ☹️
We really need to stop putting money/time/effort into these.
Reminder for the new academic year :
R/Rmarkdown is much easier than you think.
How? JAMOVI exports R syntax, lots of point & click packages, R can be easy.
Video & resources below 👇:
Will update video with ChatGPT 4 code analyzer in a few weeks.
🧑💻Beginner R/Rmarkdown workshop completed 🧑💻
Simple powertools making R easy with copy-paste. Get you started, overcome scary transition to R.
No coding required to do amazing things, get started with point & click, build skills, learn, improve.
Video:
Accepted for publication at Collabra:Psychology:
"The Effort Heuristic" revisited: Mixed results for replications of Kruger et al. (2004)'s Experiments 1 and 2
More info below 🧵👇
= unrealistic demands of replications & replicators =
yes it's that time again, another rejection for 1 of our best replication projects. again, from JESP, which keeps disappointing me. again, editor that for 2nd time doesn't seem aligned w/ journal/editorial policy.
[thread]👇
== Preprints: Open guides - effect sizes, confidence intervals, and power. ==
Qinyu Xiao took over guides that I started to guide students in my courses, & did a remarkable job bringing these to become top notch resources.
We would love feedback.
Feel free to use/share.
🧵👇
Great news.
Our mass pre-registered replication project (~45 projects so far) received 250000HKD (~32000USD) from
@hku_science
Teaching Development Grant to support replications next semester.
Info:
Join us:
More @
#sips2019
🌐=Reproducible Meta-analysis Registered Report templates=🌐
1. We face serious issues in reproducibility of meta-analyses.
2. We need to move towards model of
#metaanalysis
Registered Reports.
Join us in improving meta-analysis
#openscience
@RegReports
templates!
Details👇
🟢Time to vote: Will it replicate?🟢
(please retweet!)
We're again running Twitter prediction markets on
#openscience
Replications (+extensions) Registered Reports of classic findings in Judgment & Decision Making.
For each study, please vote:
Will it replicate?
Thread 🧵👇
Mass JDM pre-registered replications results are in.
I presented preliminary (!) findings of 18 projects.
Summary (see attached table):
9 successful,
5 inconclusive,
4 likely failures.
Presentation:
Prediction markets thread:
Mass replication project: Twitter "prediction markets" results are in.
Thread:
Thanks for taking part!
Summary:
12/18 sig success
0/18 sig failure
6/18 inconclusive split
Results on Dec 20th, in the meantime...
What do you make of these predictions?
New preprint:
"The value of replications goes beyond replicability and is tied to the value of the research it replicates: Commentary on Isager et al. (2024)"
My very quick take on the Isager and team's work on value of replications.
Read:
Why do open science? What is the science crisis? Why do replications?
Lecture about my journey into open-science & mass-replications project I'm heading.
Video:
Slides:
[1st video sharing, awkward & some errors. Go easy on me.]
#SIPS2019
was the best conference I ever attended. Really, my kind of people.
Had great conversations that resulted in brilliant ideas, many that led to high-potential real projects w/ decent progress in short hackathons.
Inspired, & motivated to work with teams onwards.
== Peer Community In (SIPS): Are you interested? ==
At
#SIPS2023
I've talked to many about the revolutionary Peer Community in Registered Reports.
We should consider forming Peer Community in Psychology/Meta-Science.
Anyone interested in leading/joining such initiatives?
🟢Time to vote: Will it replicate?🟢
(please retweet!)
This academic year, again, we're running Twitter prediction markets on pre-reg replications (+extensions) of classic findings in Judgment & Decision Making.
For each study, please vote:
Will it replicate?
Thread below👇
Positive note:
- Editor granted us R&R despite angry original authors. I appreciate that.
- 2nd reviewer emailed editor commenting on inappropriateness of original author review.
- May lead to further work & maybe adversarial collaboration (if I can overcome current feeling).
= Student early career researchers to follow =
To wrap up 2020, I want to recommend several student early career researchers I've been fortunate to work with in the last year, that I think you should follow.
They are doing terrific work and are dedicated to open science.
= Our perspective regarding Heyman and Ariely (2004) expression of concern =
Psychological Science issued an expression of concern regarding H&A2004 mentioning our team as raising the concerns.
I wrote the following regarding our role and perspective:
🔬 Recruiting peer reviewers for mass replications project 🔬
This year, again, we'll be running replications & extensions of classics in JDM & social psychology.
UG students write Registered Reports during the semester.
We have a replication targets list, details below.
A major re-vamp of the "effect size and confidence intervals" guide.
Matthew took this to a whole new level, above & beyond my expectations, building on our fairly simple guide to now cover great many things about all that's effect sizes related. 👏👏
Join us! help us improve.
Big 2024 announcement! The “Guide to Effect Sizes and Confidence Intervals” had a pretty substantial update and is slowly becoming the most comprehensive resource on effect sizes and confidence intervals. Here is a thread of some of the new additions 1/5
Accepted for publication at Collabra:
"Revisiting and Rethinking the Identifiable Victim Effect: Replication and Extension of Small, Loewenstein, and Slovic (2007)"
Failed replication & ROBMA re-analysis of a published meta, concluding no effects.
More info below 👇🧵
= Upcoming
#openscience
online talk @ Uni of Vienna =
"“Check me, replicate me” pledge: Promoting a science of collaborative replications, assessments, and corrections"
Thursday, March 31, 2022; 3pm CET
Email colloquium.psychologie
@univie
.ac.at for Zoom link.
[Thread] I enjoy running classic JDM experiments on students in the classroom.
Below I'll try & share some of what I run.
All materials and slides shared on:
Looking for more? See previous years JDM and Fundamentals here:
Want high quality
#openscience
feedback on your
@RegReports
? Try
@PCI_RegReports
In recent weeks, we received R&R feedback on 2 PCI-RR submissions. Open-science supportive identified peer-reviewers, with some of most positive, constructive, & detailed feedback we ever received.
= Beginner R/Rmarkdown workshop =
Will give online Zoom workshop
"Practical hands-on beginner R/Rmarkdown workshop: R is easier than you think, time to move to R"
3-6pm HK time (GMT+8) | Oct 29 (Friday)
Poster attached. All welcome.
Registration:
Going beyond the rejected grant application second year straight - which I didn't really have expectations of getting, there's a clear message here:
Doing replications/open-science is harmful for careers of early career researchers.
Is this the message HK's RGC wants to send?
I gave a workshop at
@HKUniversity
on
@jamovistats
/
@JASPStats
/
#R
this week.
Presentation:
Materials:
JAMOVI/JASP/R collaborative manual:
Replication guide collaborative manual:
Ever ran 22 studies in 2 weeks?
This morning I concluded data collection for our mass pre-reg replications + extensions this semester ().
Setup explained in the slide attached.
Quick insights about the process:
🟢Time to vote: Will it replicate?🟢
(please retweet!)
We're again running Twitter prediction markets on
#openscience
Replications (+extensions) Registered Reports of classic findings in Judgment & Decision Making.
For each study, please vote:
Will it replicate?
Thread 🧵👇
= Help: Meta-analysis coder needed as coauthor =
We have an experimental Meta-Analysis Registered Report on action-inaction that received a Stage 1 in-principle from Peer Community in Registered Reports.
🤜Looking for coder coauthor to join team to help us code.🤛
Details 👇
🟢Time to vote 2024: Will it replicate?🟢
(please retweet!)
We're again running Twitter prediction markets on
#openscience
Replications (+extensions) Registered Reports of classic findings in Judgment & Decision Making.
For each study, please vote: Will it replicate?
🧵👇
A decade of open-science and science reform, replications are much more mainstream now, right?
Wrong.
See Beth's preprint below.
My summary slide is attached (Beth's sent by Simine).
Have we improved? maybe, not by much.
Sometimes feels like 1 step forward 2 backwards
👇
We’ve spent the last decade talking a lot about replications, but to what extent has this talk translated to a tangible change in how often replications are published in psychology journals? (1/9) /w Katherine Lee
@SchiavoneSays
@mijkenijk
@siminevazire
Published at Meta Psychology:
"We are all less risky and more skillful than our fellow drivers: Successful replication and extension of Svenson (1981)"
The result of a fun workshop I ran with the wonderful Linköping Sweden JDM team.
More details:
👇🧵
New preprint: "We are all less risky & more skillful than our fellow drivers: Replication & extension of Svenson (1981)"
Fun little replication with amazing
@jedilab_liu
:
@linakoppel
@DanielVastfjall
&
@tinghog
Preprint:
OSF:
Today, we received decision letter from SPPS. I share this because these are very rare & so I feel
@EmirEfen
@PRASAC
& students deserve big congrats on work they did here. Balances a bit other hostile reviews we recently received on replications.
Hope for more of those to come
Thought I'd share something about my academic path.
I graduated my PhD late 2014. I've been on academic path since ~2009. The paper retweet below is 1st preprint I've been invited to fully collaborate on.
The 1st.
All projects till now I've had to initiate & lead on my own.
"Exceptionality Effect in Agency: Exceptional Choices Attributed Higher Free Will Than Routine"
w/ Fillon, Lantian, & N'gbala.
Preprint:
Great to finally see this out.
Ties well with our previous "bad is freer than good" ()
Hong Kong had its first ever Judgment and Decision Making (JDM) day yesterday, arranged by
@davidhagmann
, hosted at HKUST Central with panoramic harbor views.
30+ enthusiastic early career JDM scholars. Great to see Hong Kong universities come together.
In their latest paper, Nadia Adelina and Gilad Feldman (
@giladfeldman
) replicated and extended studies from Pronin and Ross (2006) on perception of past and future selves. A thread 👇👇👇1/
Time to vote: Will it replicate?
This year, again, we're running Twitter prediction markets. This semester we completed 12 pre-reg replications (+extensions) of classic findings in Judgment & Decision Making.
For each study, please vote:
Will it replicate?
Thread below👇
WOW. Noticed this in new preprint by
@peez
:
"Thanks Results Reproduction (R-Squared) service of Cornell Institute Social & Economic Research (CISER) for proofreading our data & code"
Leading to:
How do I convince my uni to do this?
(
@willhayward_nz
🤓🔬)
In conclusion - perhaps the best of all:
"Mass replications may not be a worthy research direction for an assistant professor, who is under time pressure for earning tenure and should focus on
building/testing theories and developing expertise in a subject area."
😳😮🤦♂️
HKU psychology hiring 3 faculty:
Associate/Assistant Prof. in Clinical Psychology - Aug31
Professor/Associate/Assistant Prof. in Educational/School Psychology - Aug31
Associate Prof. (Research): June30
👇
2nd episode on issues with academic research in Freakonomics Radio podcast. A lot of important highlights
Link:
I'll list a few that I thought stood out:
🧵👇
Now online:
"Fallibility in Science: Responding to Errors in Work of Oneself and Others | Dorothy Bishop"
Interesting chat with Dorothy Bishop. 🙏
Video:
I care greatly about this topic, & am looking to collaborate with others on addressing it.
👇🧵
🤔As humans we are all fallible. When it comes to research, how should we deal with errors in the work of our own and others? ✨
Join us in our next “Meet the Authors” Reading Club event with Professor Dorothy Bishop
@deevybee
to find out more about fallibility in science!
Every time I do copyediting proofs from publishers I end up spending more time correcting their mistakes than would take me to format things on my own. Takes hours to check things.
Headings are never right, oversights, many small annoying mistakes. Such a frustrating process. 🤦♂️