Do you want to do a psychology experiment while following best practices in open science? My collaborators and I have created Experimentology, a new open web textbook (to be published by MIT Press but free online forever).
Some highlights! 🧵
People are testing large language models (LLMs) on their "cognitive" abilities - theory of mind, causality, syllogistic reasoning, etc. Many (most?) of these evaluations are deeply flawed. To evaluate LLMs effectively, we need some principles from experimental psychology.🧵
How do we compare the scale of language learning input for large language models vs. humans? I've been trying to come to grips with recent progress in AI. Let me explain these two illustrations I made to help. 🧵
Wordbank is an open database of children's vocabulary development across languages, archiving parent reports of what their children say.
Today we are updating Wordbank, adding 10k children, 9 new languages, and data on bilinguals.
Everyone makes mistakes during data analysis. Literally everyone. The question is not what errors you make, it's what systems you put into place to prevent them from happening. Here are mine. [a thread because I'm sad to miss
#SIPS2018
]
What does it mean for a large language model (LLM) to "have" a particular ability? Developmental psychologists argue about these questions all the time and have for decades. There are some ground rules. 🧵
A thought on grad advising. When I was a second year, an announcement went out to our dept. with the abstract for a talk I was giving in the area talk series. A senior faculty member wrote back with a scathing critique (cc'd to my advisor,
@LanguageMIT
). /1
How do we use methods from developmental psychology to assess AI models? My comment, "'Baby steps' in evaluating the capacities of large language models" is now out in Nature Reviews Psychology:
Errors in data analysis code are a major problem for scientists and for science. Two practices that can help you avoid them (or track them down more easily) :
1. never copy and paste code, and
2. never write a code block longer than your screen.
[thread]
Can a large language model be used as a "cognitive model" - meaning, a scientific artifact that helps us reason about the emergence of complex behavior and abstract representations in the human mind? My answer is YES.
Why and under what conditions? 🧵
What is "the open science movement"? It's a set of beliefs, research practices, results, and policies that are organized around the central roles of transparency and verifiability in scientific practice. An introductory thread. /1
Coming soon: the Open Encyclopedia of Cognitive Science, an open access reference work devoted to the study of the mind.
@asifa_majid
and I are co-editors in chief – articles will be posted starting in 2024.
The first ManyBabies project, "Quantifying sources of variability in infancy research using the infant-directed speech preference," has been accepted at AMPPS!
This 67 lab (!) paper shows the power of collaboration to advance methods and theory!
The world is on fire. But if you are teaching undergrads how to read a scientific paper for the first time,
@aenordmeyer
and I made a handout to help your teaching life.
Choose an easy paper and walk through exercises together as students read in class.
CHILDES is the premier resource for studying transcripts of children's language.
We created , a reproducible, versioned interface.
Now childes-db has a 2020 update, with 1) updated R API, 2) new Python API, 3) phonbank, and 4) bugfixes.
Announcing the launch of the Open Encyclopedia of Cognitive Science: ! OECS is a freely-available, growing collection of peer-reviewed articles introducing key topics in cogsci to a broad audience of students and scholars.
Congrats to everyone who applied to the NSF GRFP! For many it’s their first independent grant application, and people pour their hearts in it. Well done, regardless of outcome!
(My first GRFP got abysmal ratings. “Broader impacts: none” was one of the nicer comments.)
My lab held a hackathon yesterday to play with places where large language models could help us with our research in cognitive science. The mandate was, "how can these models help us do what we do, but better and faster."
Some impressions:🧵
Convenience samples are ubiquitous in psychology. The field has been soul-searching for years about whether this sampling practice undermines generalizability. But convenience samples are only a problem in the presence of population heterogeneity in the measure of interest.
Developmentalists: have you been wondering if online experiments with young children are comparable to in-lab studies? Here's our new meta-analysis suggesting that overall they are!
- work by Aaron Chuey,
@veroboyce
and
@anjie_cao
I am both thrilled and disturbed that arranging vegetables into a face causes my toddler to eat them with relish. “I am eating his EYES” he screams with glee.
Metalab is a website for theoretical synthesis in developmental psychology using statistical meta-analysis. We just released a major update, now with data from >16k babies and >250 papers.
In my first summer as a faculty member, I assigned a bright intern a project to do smartphone eye-tracking (using open source libraries that already existed).
I was wildly mis-calibrated about how hard it is, but now it's been done:
Some people say grad school is a marathon, not a sprint. It's neither. It's not a race at all. It's about becoming part of - and contributing to - an intellectual and professional community.
For two years,
@mbraginsky
,
@danyurovsky
, Virginia Marchman, and I have been working on a book called "Variability and Consistency in Early Language Learning: The Wordbank Project" (
@mitpress
).
Here's our draft:
[+ a thread with a few of our findings]
ManyBabies 1 now out! Quantifying sources of variability in infancy research using the infant directed speech preference. Babies prefer babytalk the world around - but preference strength was moderated by age, language, and method.
Are you analyzing likert data as normally distributed because you don't know how to do mixed models otherwise?
Do Bayesian ordinal mixed effects regressions w/ the awesome
@paulbuerkner
brms package.
A short tutorial:
Longer:
Excited to welcome our lab’s newest research organism. Leah Okumura is doing a sabbatical from
@Wellesley
on this amazing critter. Can’t wait to look at their cells divide. Does anyone know what it is? (Answer soon)
I found myself telling this story at a conference the other day as an example of awesome advising and wanted to share it more broadly because I think about it a lot. Thanks,
@LanguageMIT
! /end
As an advocate for methodological reforms in psychology, I'm not sure what to think that the result of our work is that each journal now has an extensive and unique set of requirements for disclosures, open science, sample etc. that add hours of time to the submission process.
People believe you can just throw lots of control variables in a regression. It’s hard to overstate how common this belief is in psych and education students. This is a wonderful article explaining the complexities and risks.
Candid moments from my google history this week:
“binomial distribution wikipedia”
“how to write specific aims”
“piaget wikipedia”
“ggplot rotate x labels”
“apa format guide”
I’m also not quite there for grad school by this definition. 😂
Can large language models be used as scientific models for studying human cognition? Here's my argument they can, with Theory of Mind as a case study:
The key point: for LLMs to be good models, they must be openly accessible to researchers.
I recommend you don't preorder our book, "Variability and Consistency in Early Language Learning: The Wordbank Project" – it's free online: .
But I'm still really excited to see it up on Amazon and love reading blurbs from three of my intellectual heroes.
Undergrads, please join us for a 8-week paid internship at Stanford CSLI in our
@NSF
-funded research experience program in cognitive science!
Interns do mentored research and learn about language, cognition, AI, and more as part of a warm, fun community.
Save time, avoid copy-and-paste errors, and make beautiful documents! My tutorial from
#icis18
on using RMarkdown for writing reproducible research papers:
Here's the handout with exercises and instructions:
I'm teaching "Language Learning: A Data Driven Approach" at the LOT winter school in Amsterdam this week. Students can learn how to play with data from Wordbank, childes-db, and Peekbank using R and the tidyverse!
All slides and exercises available here:
3. When you claim an LLM has X (ToM, causality, etc.), you are making a claim about a construct. But you are testing this construct through an operationalization. Good expt'al psych makes this link explicit, arguing for the validity of the link between measure and construct.
Ever wondered why we are always talking about stages and discontinuities in child development, and how we could test this kind of claim? A new blogpost on this topic:
Nothing in childhood makes sense except in the light of continuous developmental change
Early life stress and adversity is related to when molars erupt. That's right - kids who have lower income and greater early life stress on average have their teeth come in earlier!
Super creative prediction by
@ally_mackey
(life history theory!):
What matters more for mixed effects models, the random effects structure (maximal vs. not), or the fitting procedure (frequentist vs. Bayesian)? A new blogpost exploring ManyBabies data.
Spoiler: It's the random effects, stupid!
Next year I'll be director of Symbolic Systems, Stanford's undergraduate cognitive science major. We're hiring a teaching coordinator to help build introductory course offerings. If you have a relevant PhD and are interested in pedagogy, please apply!
This is generally a great paper! But nearly all responses to the replication crisis suffer from the "all psychology needs X" problem. This manifesto doesn't apply to cognitive science, and it's the opposite of what dev. psych needs (highly theorized, with noisy/limited data).
New paper: we argue that the replication crisis is rooted in more than methodological malpractice and statistical shenanigans. It's also a result of a lack of a cumulative theoretical framework: &
How should we reason about the vast differences in input data between large language models and humans? Here’s my new preprint, “Bridging the data gap between children and large language models”:
I am deeply upset by this statement because it buys into the false narrative that it is somehow partisan to call out an inhumane and unjust policy on the basis of our science. I would like to see the
@SRCDtweets
leadership explain and defend this choice.
Statement of Laura L. Namy, Executive Director, Society for Research in Child Development on U.S. Policy of Separating
#Immigrant
Children from their Families at the Border:
This kind of support is what I aspire to as an advisor, so that my students can carry on doing the work they want to do, and have some measure of insulation from the (occasionally) ugly territoriality of professional life. /6
Ominous dream: I did a study whether toddlers liked running around while listening to whale song. I described it to a colleague as “my new Nature paper.” Then had the awful sinking feeling as someone asked me, “What was the hypothesis? What question were you trying to answer?”
CogSci 2020 will be virtual rather than in-person. Disappointed not to go to Toronto, but happy that we can still share our work.
@cogsci_soc
is collecting ideas for a smooth virtual conference with multiple tracks, poster sessions, and participants in many time-zones.
Syntactic abstractions emerge in neural networks trained via word prediction: cool paper by
@chrmanning
et al. Supports early speculations by Jeff Elman & highlights that we need more work connecting modern neural nets with psycholinguistic data!
2) Standardize your workflow. If you do things consistently, you will be less likely to make new, ad-hoc errors that you don't recognize. For me this has meant learning - an amazing ecosystem for R data analysis.
My tutorial: .
Tiny language acquisition observation of the moment. Jonah at 20 months has no productive vocab (good receptive though). Yet he grunts iconically for size: high pitch “eh” for small ducky, low pitch for big ducky. Nice cross dimensional iconic maping
@asifa_majid
@DingemanseMark
?
Ever wondered how kids learn logical language - words like "or", "no", or "if"? Maybe they are learned by generalizing social concepts like "offer", "rejection", and "threat."
Some thoughts on this "social bootstrapping" hypothesis in a new blogpost:
My lab, the Stanford Language and Cognition Lab (), is looking for a new Research Coordinator to start this summer. Lots of ways to get involved in research and learn new skills/methods in a fun, supportive environment!
Flip it around: it'd be super weird to approach a person on the street, ask them a question from the computer science literature (say, about graph coloring), and then based on the result, tweet that "people do/don't have the ability to compute NP-hard problems"!
So, principles:
Child Development, the flagship journal of
@SRCDtweets
, has announced a Registered Reports special section (with me and
@syeducation
as section editors).
This is just one of several steps that Glenn Roisman (new EIC) is taking to embrace transparency and reproducibility.
Introducing LEVANTE, a project for measuring children's developmental and learning variability around the world! LEVANTE provides a measurement platform with data flowing to an open repository. Call for proposals coming soon from
@Foundation_JF
.
OK, it's Friday night. Time to yell into the void about journals that limit the number of citations you're supposed to have. This should never be a thing.
Very nice review of the literature on human statistical learning. The surprising conclusion? Chunk-learning, rather than transitional probability tracking, best describes human performance.
In lab today we read Fei Xu (2019), Psych Review, "Towards a rational constructivist theory of cognitive development." This is an ambitious theoretical paper that tries to synthesize core knowledge with Bayesian modeling and the "theory theory."
The mystery of children’s first words. Great story in
@TheAtlantic
by
@michaelerard
- featuring my work on using larger datasets to study children’s language with Wordbank (), and also a guest appearance by my son Jonah!
Linking Quality and Quantity of Parental Linguistic Input to Child Language Skills: A Meta-Analysis (Anderson et al., 2021)
Quality > quantity, naturalistic > in-lab as predictors of outcomes.
Worth noting these estimates are still non-causal!
Earlier this year, Child Development called for Registered Reports as part of a special section edited by
@syeducation
and me (along with Glenn Roisman, EIC):
We just read >110 letters of intent and invited ~25. Here are some thoughts from that process.
Come work with me as a postdoc on the ManyBabies project, a collaborative framework for replication, best practices development, and theory building in developmental psychology!
Why use ideas from experimental psychology? Well, ChatGPT and other chat LLMs are non-reproducible. Without versioning and random seeds, we have to treat them as "non-human subjects."
I don't know how interested
@OpenAI
and
@sama
are in making GPT-4 into a research tool, but I think the first thing we'd need to do is establish reproducibility.
To this end, we need:
1. build numbers (so we know the version)
2. random seeds (so we can reproduce random behavior)
OK, so turns out to be the best part of the cogsci poster sessions! I actually "ran into" a friend while wandering between posters. Highly recommend that folks do their poster Q&A on the platform,
@cogsci_soc
(put poster ID on your name)!
#CogSci2020
I edited a special issue of Infant Behavior and Development on Replication, Collaboration, and Best Practices in Infancy Research:
Lots of great papers! Here's my introduction:
In sum: of course we should be afraid of errors! But don't *just* be afraid. Accept that you *have* made errors - and will make more. We all do. Act to put systems in place that help you catch and correct errors before they enter the literature. [end]
Undergrads! Join us for an 8-week paid internship in the 2022
@StanfordCSLI
Summer Program, an
@NSF
REU site.
Interns do mentored research and learn about language, cognition, AI, and more as part of a warm, fun community.
1. An evaluation study must have multiple observations for each evaluation item! It must also have multiple items for each construct. If you have one or a few items, you can't tell if it's the idiosyncrasy of items that led to the observed responses. See:
We need an ethical framework for
#OpenScience
decision-making! How can we address risks while appropriately valuing openness - for data, code, materials, and research reports? A thread arguing that the key is balancing who bears the risks and rewards.
Evaluating scholarship via metrics like H is far inferior to reading papers in depth and discussing them with colleagues. Academic evaluation *must* be incentive compatible: If we want good scholarship then we have to hire and promote on that basis, rather than via bad proxies.
On the "stats in psycholinguistics debate" (aka "back to ANOVA?"), I get the frustration. It's totally annoying to have to work to understand the vagaries of Stan rescaling and convergence when all you want to do is your 2x2. On the other hand...
Did anyone get taught in grad school that main effects are not good for theory building and that interactions are critical for revealing mechanism? I definitely got told this but I never really understood it deeply. Anyone know a citation/rationale for that claim?
We're pleased to announce a new
@Many_Babies
project, "A Unified Approach to Demographic Data Collection for Research with Young Children Across Diverse Cultures":
We want data to be FAIR - findable, accessible, interoperable, and reusable. Big piles of data in generic repositories are findable, but no one reuses them. To promote *reuse* we need... domain specific data repositories (DSDRs).
New blogpost!
Really excited to see this work coming to fruition! Jess Sullivan and
@AmyPerfors
suggested doing a longitudinal head cam study of their own kids seven years ago and now these super-rich data are in their way out into the world!
Tools for open science! Join us today to learn about R, RMarkdown, the tidyverse, and the open science framework. All materials here: - stop in whenever would be useful (don’t worry if you didn’t register).
#ICIS18
Consistency and variability in word learning - updated preprint on predicting what words are harder and easier for kids to learn. Still the most exciting work we've done with Wordbank so far (by
@mbraginsky
,
@danyurovsky
, Marchman, & me).
I know the world is burning right now, but it’s still nice that
@cogsci_soc
keeps taking small steps to make our community more inclusive. Thanks
@asifa_majid
for your leadership on this one.
We're pleased to announce that this year we will be offering grants of up to $500USD to help with family care expenses incurred while folks attend our annual conference. For eligibility info and more details on the CogSci Family Grant Program visit:
New Stanford Encyclopedia of Philosophy article on "Reproducibility of Scientific Results" - a comprehensive meta-scientific and philosophical discussion of the issues!
@StanfordCSLI