genuinely shocking to me that study after study shows that social scientists are no better than regular people at high-level social science predictions: whether studies will replicate, which "nudge" interventions work, how social attitudes will change
a thread of examples:
nick patterson: child chess prodigy. math phd at cambridge. code-breaking for british intelligence. personally invited by simons to join renaissance tech hedge fund. ancient dna pioneer inventing much of modern popgen toolkit. massive deformed head. cartoon character-ass life
so the purported genetic basis of no-pain scottish lady is:
1) microdeletion downstream of gene FAAH
2) plus a common (25%) activity-reducing variant in FAAH
~1/12k europeans have this combo. this is common enough that if it caused no-pain syndrome we would likely know of it:
okay the no-pain scottish lady is on my feed enough that i should maybe read the paper to see how much i believe the putative genetic cause
after the low-sleep genes debacle i'm skeptical of attempts to identify causative mutations in 1-3 individuals with a weird phenotype
social scientists *were* substantially worse than regular people at predicting which interventions would make people more likely to get a covid vaccine:
(see pic, which also includes a rather rosy description of the next study in this thread)
37 behavioral scientists designed a
23 condition megastudy testing different sets of
1-2 text messages to boost vaccinations among
689,693
@Walmart
pharmacy customers
430 forecasters tried to predict what worked
NOW our results are out in
@PNASNews
... 🧵
(to be clear, we're not even talking about superforecasters here, just regular joes)
social scientists, applied and academic, were no better - if anything, possibly worse - than regular people at predicting which interventions would increase gym visits:
None of the groups made accurate predictions about what behavior change methods work! Correlations between estimated treatment effects and observed ones were
• ordinary people: r = 0.25, p=0.07
• professors: r = −0.07, p=0.63
• practitioners: r = −0.18, p=0.19
a blunt analogy: if a bunch of people went around getting "metereology" phds, and calling themselves "metereologists", but it turned out they were no better than random people at predicting if it would rain tomorrow, i would be annoyed. this is how i see a lot of social science.
nowadays when i hear of a new academic fraud i'll sometimes email their institution about it. "this is bad, i think less of your institution now" etc
if cancel culture taught us anything, it's that 5 or so emails in the right inboxes can provoke strong responses by organizations
update on the scottish no-pain FAAH story: i decided to check myself whether FAAH is associated with pain and found no signal.
first, i ran associations in 430k sequenced british people in uk biobank on *all* 30k variants in and near FAAH, for 60 pain + mood phenotypes:
regular people had larger absolute error (as is often the case) when predicting the effect of incentives on performance of a boring task, but their rankings of the interventions were as good as social scientists':
social scientists were no more accurate than regular people at predicting or *retrospectively assessing* the social consequences of the covid-19 pandemic:
Social scientists were no more accurate than lay people in predicting or assessing social consequences of COVID-19; estimates of the magnitude of change were off by more than 20% and <1/2 accurately predicted the direction of changes
#SocSciResearch
Laypeople Can Predict Which Social-Science Studies Will Replicate Successfully
TL;DR: If a study clashes with commonsense, you should probably side with commonsense (at least till it's been properly replicated).
in a follow-up paper to the one above, social scientists were usually no better than regular people at predicting how general social attitudes would change over time:
Very excited to share this new paper out at
@NatureHumBehav
!! w/
@psywisdom
@cendripetalfrce
Scientists' forecasts for societal change (polarization, life sat, biases++) are not more accurate than statistical models. More accuracy w/ expertise, interdisciplinarity.
🧵⬇️(1/9)
in estimating gender bias in hiring over time, regular joes' predictions were more extreme than academics' - again, typical - but correlations with real values were very similar, and both made the same mistakes, e.g. thinking there is still bias vs women:
On the trajectory of discrimination: A meta-analysis and forecasting survey capturing 44 years of field experiments on gender and hiring decisions
“[B]oth scientists and laypeople overestimated the continuation of bias against female candidates.”
short-sleep genes, no-pain genes, blind-genius genes...
funny how this exact study recipe - try to pin some wacky phenotype in a small pedigree on a single gene using incomplete genotyping - is responsible for a large proportion of the viral-but-wrong science facts on my feed
a recurring result in these is that expertise level (undergrad vs professor) doesn't matter for accuracy, expert field (economics vs marketing) barely matters - training might make you better at publishing papers in a field, but not (clearly) better at giving actionable advice
very cool paper. permafrost doesn't just preserve DNA - it preserves structure too, including chromatin loops, barr bodies, inactive/active compartments, and allows a woolly mammoth genome assembly even with short aDNA fragments
come for the science, stay for the charming icons
It’s finally out! 🥳 Today
@cellcellpress
we report non-mineral fossils of ancient chromosomes in skin from a woolly mammoth that died in Siberia, 52,000 years ago.
🦣💨
Don’t miss our thread below! 🧵👇🏽
(not grinding a political axe by mentioning the above, that's just what the study says!)
for fairness, here are some papers concluding that social scientists are better than laypeople at various prediction tasks:
a post-script on this FAAH business. why did i, a conflict-averse neurotic, embark on this very public crusade against a single research finding? here are some reasons why:
@leonvarkalis
@sarahzhang
@cecemoore
i/o and the quoted guy are totally wrong. 1st-degree incest may be more common in some ethnicities but uk biobank gives no evidence for this. ~5/6 of cases found there are genetically white british. i have run these analyses myself, feel free to grill me on this
if i had a billion dollars i would build a glistening 500-person facility to do the corn long term selection experiment on the stupidest traits. horses with really big ears. gradually breed worse and worse smelling goats. select capuchin monkeys for jenga ability. and so on
New: Read the story of a decade-long propaganda campaign by the Forrest Gump of the internet—a Wikipedia admin who was once Yudkowsky’s strongest soldier—set against the backdrop of the collapse of the semi-unified Internet ethos of the ‘90s and ‘00s
1) long-term effects of rct interventions - note studies were mostly in africa (and afghanistan) with predictors in the west:
2) which interventions make people less likely to click "anti-democratic" options in online surveys (...)
case in point: remember those "low-sleep genes"? they were identified using a similar methodology... but none of the variants that could be tested actually replicated in a larger study:
the ancient DNA hobbyist phenomenon is wild. imagine 1000s of online weirdos obsessing over your field for 15 years, most are insane & racist, but they jury-rig new tools that sorta work, and the top 3 are better than all pros at evaluating claims & some get hired as researchers
moral of the story: even if you bend over backwards to make inclusion a core principle of your work, if you commit one faux pas in presenting it, the people you're trying to please will accuse you of facilitating hate crimes and demand that you retract your paper
the microdeletion is the money variant here. in the patient, it's 8kb long and 4.7kb downstream of FAAH
but in the gnomAD database, i see a deletion basically identical to this carried by ~1/6,000 europeans (red arrow), and an even bigger one carried by ~1/4,000 (green arrow)
okay the no-pain scottish lady is on my feed enough that i should maybe read the paper to see how much i believe the putative genetic cause
after the low-sleep genes debacle i'm skeptical of attempts to identify causative mutations in 1-3 individuals with a weird phenotype
unrectified science has consequences. futile initiatives. phds spent chasing phantoms. whole labs, entire subfields, internalizing a false principle from a wrong result and building upon it a dozen hopeless babel towers of work. the vast waste of it all horrifies and animates me.
3) mask-wearing nudges on democrats and republicans. but - i plotted the results, and it seems most laypeople didn't fully understand the (poorly worded) prediction question and mostly assessed general dem vs rep mask attitudes, not intervention effects..
if i've helped even slightly to course-correct these doomed dollars and man-hours, it will bring me more contentment than most things i can imagine.
this is how i actually think! i do it despite the attention making me nauseous with anxiety. i'm a strange, intense little man.
repeat after me: you need more than one case. you need a bigger pedigree for mapping. you need whole-genome sequencing. you need to stop pretending that you know in advance which genes are important. you need to check your results in some of these huge biobanks we have now.
i think that if this caused "no pain, no anxiety ever syndrome" in 1 in 12k europeans, science would probably have given it a name. it would run fairly reliably in families. the cause might not be known, but its existence i think would be.
for a plausible one. (here, they had to resort to a combination of variants!) (nb: they didn't even fully sequence her...)
i'm still leaving some chance that this is real, but hey: let's analyze the ~40 people in uk biobank who likely have this combo. prove me wrong!
sadly and ironically, scott's old post on 5-httlpr - a skewering of candidate gene researchers who built "castles in the air" on a foundation of poor statistical power, ignorance about genetics, and general wishful thinking - is relevant here:
identifying the genetic cause of a condition you see in very few people can be done, but it's hard. people carry many, many rare variants. in this kind of study you genotype a person with a weird condition, see 1000s of candidate mutations, and try to come up with a just-so story
@jasoncbenn
@bryan_johnson
oh that candidate gene study failed to replicate
anecdotally I'm homozygous for the "short sleep" version of dec2/rs121912617 and usually fully need an 8 hours
so, show's over. there's nothing here. a lab which *still* hasn't learned that candidate gene-style thinking doesn't work misled themselves, confabulated a biochemical mechanism, and some poor schmucks set up a damn foundation based on it. it sucks. but let's learn from this!
i gotta say that when i found that the purported cause was a combination of two mutations, and moreover these being 1) a very common variant and 2) a pseudogene deletion, i became skeptical. yes, there are compound heterozygote phenotypes, but they're unusual.
so, i once compared the uk biobank ancestry cohorts defined by 2 research groups. one drew around umap blobs, the other used a random forest trained on reference samples.
seems that these methods are broadly viewed as fickle and dumb, so the extent of agreement may surprise!
the other variant. this is very common - 1 in 5 europeans carry it. this is one of those fun missense variants in an enzyme which has loads of p to the minus bazillion associations with metabolite levels on gwas catalog, but no clear effects on anything else, health or otherwise.
getting emails is a strong signal to the admin that people do in fact notice and care about the issue. especially important if senders have highly legible status and influence (professors, grant-makers, etc)
unlikely that *my* emails will move the needle much but hey, i'm trying
critical support for pachter here in publicizing this: clear, undeniable fraud in the work of mv srinivasan.
annoyingly, srinivasan basically won: he's 75 now, career basically over, enjoyed 22k citations and lots of nice awards. hope it was worth it.
Aristotle was the first to notice honeybees dancing. In 1927 Karl von Frisch decoded the waggle. How it works was "explained" by MV Srinivasan AM FRS in the 1990s. Except
@NeuroLuebbert
found his papers are junk. A 🧵 about her discovery & our report: 1/
@DilettanteryPod
there are multiple stories of inuits washing up in northern scotland, at least some probably true... damn, i actually find this somewhat plausible
@CarlosEAlvare17
for some domains, quite likely. i apparently felt enough social pressure to start defending myself in the thread when i mentioned that there may no longer be a bias against women in hiring decisions!
(even the v similar microdeletion might not be identical to the patient's; these microdeletions might not have the same effects as the patient's. but if the region *were* important for pain i think you'd see *some* pain-related effect, especially with the alleged 2nd ingredient:)
none reach genomewide significance, nor the relaxed threshold i set. even if go very relaxed and look at any old association with p < 1e-4, no marker appears more than once across phenotypes. (a general pain-reducing variant should reduce multiple kinds of pain).
@billyhumblebrag
might reach out to the lab that published this case study, yeah. i just feel quite awkward about it all. it's one thing posting this haughty thread as an anonymous coward - another thing entirely to actually go directly to these folks
@odoreida
thanks. if i were forced to work with social scientists i'd probably choose economists. good at math, at least try to see if their results are robust. just gotta steer them away from instrumental variable or regression discontinuity analysis
the word "mutation" in genetics is surprisingly imprecise, and can refer to
- the event which changes a genetic sequence
- the resulting sequence
- the general concept of genetic sequences changing
i'd prefer "mutation event", "mutant allele", "mutation" respectively
eric and sonia might actually find a cure before her fatal familial insomnia manifests. if they do it will be one of the most incredible stories in human history. it really makes me emotional
Introducing CHARM: a new epigenome editor to methylate DNA at the promoter of a targeted gene.
Our lab's collaboration with
@JswLab
's
@EdwinNNeumann
&
@TessaBertozzi
shows deep silencing of brain PrP
Paper:
Blog:
transpires that a new paper relevant to my interests was authored by someone i dated in college but absolutely blew it with, in a really shameful manner
non-negligible chance that we end up as co-authors at some point. trying to convince myself that this is "funny"
i'd seen the preprints so not breaking news for me, but still find it bananas that some people's y chromosomes are almost twice the length of others'. and extent of large-scale structural variation is wild. phenotypic impacts?
@Nature
And this is the second article in
@Nature
reporting 43 Y chromosomes from men that lived over a range pf 183.000 years, hence revealing the relevant differences in sequences and structure that have occurred recently during evolution of this chromosome.
Really interesting paper. Not sure what to make of this chart. Leftmost 6 points are well below where I would've put subsistence, so I'm worried about measurement error. And then of course just tough questions of interpreting these comparisons across contexts.
witten family:
- ed: genius physicist
- matt: tv writer and novelist
- jesse: law partner
- celia: md/phd
is this (laudatory) the most jewish family of all time?
assuming that these variants are independent of each other (probably not, but probably not strongly linked), the probability of a european carrying the allegedly causative combo of 1) one of those microdeletions, plus 2) the common missense variant, is ~1 in 12k.
@imperialauditor
@leonvarkalis
@sarahzhang
@cecemoore
e.g. the 95% (wilson) ci for 2 cases in 10000 is between 1/18232 and 1/1372. you would need a completely massive survey or observe an absurdly high rate in one group to confidently say "it's highest here".
@eyeslasho
you are completely wrong about this by the way. feel free to ask me questions as technical as you like about this kind of inbreeding analysis.
@leonvarkalis
@sarahzhang
@cecemoore
i/o and the quoted guy are totally wrong. 1st-degree incest may be more common in some ethnicities but uk biobank gives no evidence for this. ~5/6 of cases found there are genetically white british. i have run these analyses myself, feel free to grill me on this
there are ~100 people(!) carrying this "genotype" in the 430k sequenced brits. great! but - i find no difference in their reported rates of various kinds of pain compared to everyone else. below i restrict to variables where i should have enough cases to find an effect:
it’s funny: *on the day of their clinic visit* 1/3 of the carriers report being depressed or anxious, and 2/3 report being in some pain or discomfort. >1/2 report chronic pain that's lasted >3 months. (nb: a random 1/3 of people were asked to complete a pain questionnaire.)
@imperialauditor
@leonvarkalis
@sarahzhang
@cecemoore
hard to quantify because repeated cousin marriage and some 2nd-degree pairings can genetically "look like" 1st-degree offspring - can classify these with some accuracy but not perfectly. plus 1st-degree is so rare that estimates are noisy for most subgroups -
for context someone actually looked to see if the various alleged low-sleep associations replicated in uk biobank and surprise!!! they completely didn't
so these guys definitely feel pain. let's check - have i screwed all this up? would i even detect a real signal? yes: for example, the common variant in the alleged scottish no-pain combo has a p<3e-80 association with blood levels of a protein.
a few carbon dates of barley seeds etc in the faroes suggest that the islands were settled 300-500 years before the vikings arrived - a minor revolution in archeology. but the "marine reservoir effect" makes carbon dates look... about 400 years older than they should. suspicious
was reading a Nature paper once and found a big spreadsheet error in the raw data. thankfully it didn't alter the conclusions, but I emailed the authors + a correction came out a few months later
huge props to the authors, but sad reminder that reviewers do not check this stuff
Alphabetical order mismatch and 52 of 78 neighborhoods had wrongly merged data. I spend a lot of time teaching advanced inference methods, but boring research data management remains the most essential skill. And that includes auditing for merge mistakes.
i also created a fake "genotype" which indicates whether a person carriers both of the mutations allegedly causing scottish no-pain syndrome. (i think their proposed mechanism doesn’t even need these variants to be on different chromosomes, so any old carriers should work).
this feels slept on:
height + education have genetic correlation ~0.2 in the general population. but it's ~0 within siblings, as the correlation is induced largely (imo) by assortative mating on "good traits" generally
"good trait" assortment has (imo) been going on a long time
@richardfuisz
i wonder if this takes into account variation in mutation rate across the genome? maybe some especially stable loci don't have an alternative allele just yet. (and i suppose this is just snps - obviously all the possible indels/microdeletions/inversions haven't happened yet!..)
like one of those victorian british families where the siblings are 1) the archbishop of canterbury, 2) some colonial governor who described 400 species of wasp, 3) a racist statistician, and 4) a painting prodigy who died at the age of 22
happy that this thread is getting liked and shared by social scientists and policy people - exactly the right audience for effecting change
less sure about the likes from dissident right guys called "breast milk enjoyer"
genuinely shocking to me that study after study shows that social scientists are no better than regular people at high-level social science predictions: whether studies will replicate, which "nudge" interventions work, how social attitudes will change
a thread of examples:
just for good measure i ran associations for just the FAAH-OUT deletion, not the combination genotype - i think they claim that this should reduce pain sensitivity too. again restricting to well-powered variables, carriers don't show any difference in reported pain rates.
uk biobank/cern/webb for everything. separation of data generation and analysis. well resourced (inter)national bodies generating vastly richer data than any individual lab could, with little incentive to cheat, data effectively available to all. extremely successful model
My (and their) point is that there is positive relationship between the average sex difference on a (sub)test and its associated variance ratio. In short, the greater male mean advantage, the greater male variance advantage (and vice versa).
was benchmarking a pipeline and kept getting very similar results across different parameters. weird - but by eyeballing plots could see that some parameters performed slightly better
then realized that i'd scrambled the labels - the "small differences" were pure noise. humbling
update: i looked into FAAH myself by running about 2 million regressions and didn't find any associations between it and any kind of pain i had measurements for
update on the scottish no-pain FAAH story: i decided to check myself whether FAAH is associated with pain and found no signal.
first, i ran associations in 430k sequenced british people in uk biobank on *all* 30k variants in and near FAAH, for 60 pain + mood phenotypes:
this paper is poignant to me in a pathetic way, as i have an elaborate fantasy where i sequence my genome, find that my psychological failings are due to one lof variant, and do embryo selection to purge it from my bloodline, which lives happily ever after
2 hours after i posted this, a paper came out saying that genetic studies of educational attainment have been strongly affected by multi-generational assortative mating (and similar processes)
this feels slept on:
height + education have genetic correlation ~0.2 in the general population. but it's ~0 within siblings, as the correlation is induced largely (imo) by assortative mating on "good traits" generally
"good trait" assortment has (imo) been going on a long time
have to respect on some level that musk, unlike other billionaires, is actually harnessing wealth and status to greatly increase reproductive success. very trad, very tribal bigman-pilled.
@HumanVarieties
are you really calculating correlations of subgroup averages? lmao
this exercise essentially just shows that iq and sat have some detectable correlation. if you simulate barely-correlated variables with tiny subgroup differences, you get r>0.95 correlation between subgroup means
@adrusi
buckyball is 5mm in diameter. they seem to pack as a simple cubic lattice, so each requires 1.25e-7 m^3. couple's dome tent seems like a hemisphere with 2.5 m radius, so volume about 33 m^3. i get 250 million; would have guessed order of 10^7
another benefit of centralized, open-access data collection in science is that most analysis fraud becomes untenable
e.g. imagine how stupid it would be to fake a uk biobank gwas hit - dozens of other groups can and probably will easily attempt a replication
uk biobank/cern/webb for everything. separation of data generation and analysis. well resourced (inter)national bodies generating vastly richer data than any individual lab could, with little incentive to cheat, data effectively available to all. extremely successful model
a lot of the coolest research is unlocked by getting much better data - order of magnitude larger than any previous dataset, or measures something totally novel
and when you're the first to work with great data, you don't even need to be that smart to get cool, novel discoveries
current workplace is an inspiring off-white pill: turns out that with good academic culture and good data, your staff don't even need to be that smart to do good work
my extended family clearly shows that people vary a lot in emotional set point and volatility
i suspect this contributes substantially to differences in fundamental quality of life. and to broader culture - incurable optimists do more in the world, are louder in the discourse
I have quite low emotional variance - I spend about 97% of my life in a narrow band of "happy". People with higher emotional variance often assume I must be mistaken or repressing my true emotions. But I think some of us are just emotionally very boring.
(i don't know how to convert that numeric id lol - possibly it's FAAH itself. this variant has known associations - see qt - so it's nice to see another one here. annoyingly i don't have measurements for the qt metabolites so can't confirm those results.)
the other variant. this is very common - 1 in 5 europeans carry it. this is one of those fun missense variants in an enzyme which has loads of p to the minus bazillion associations with metabolite levels on gwas catalog, but no clear effects on anything else, health or otherwise.
@spignal
absolutely no way the fit is this good. looked it up and indeed, the source "data" for this visualization is to a substantial extent modeled, not directly measured (page 399: )