(1/n) Can large language models simulate patients with mental health conditions?
We introduce Patient-Ψ🤖, where we integrate cognitive modeling with LLMs to simulate patients for training mental health professionals.
📎Paper link:
Happy PI🥧 day!
Can language agents🤖 learn social skills through imitation and interaction?
We are excited to introduce SOTOPIA-π🥧 , an interactive learning method for training language agents to navigate real-world social scenarios while role-playing!
Visit our poster about training social agents 🤖 at 10:30am today, at Poster Session 4!
I'm happy to talk about social intelligence, training language agents, and LLM applications to mental healthcare.
#ACL2024
Happy PI🥧 day!
Can language agents🤖 learn social skills through imitation and interaction?
We are excited to introduce SOTOPIA-π🥧 , an interactive learning method for training language agents to navigate real-world social scenarios while role-playing!
Happy PI🥧 day!
Can language agents🤖 learn social skills through imitation and interaction?
We are excited to introduce SOTOPIA-π🥧 , an interactive learning method for training language agents to navigate real-world social scenarios while role-playing!
Sharing one of my best memories from
#ACL2024
🐘🇹🇭
Spending time with (Baby) Capybaras in a cafe ☕️ !
These little ones are absolutely adorable.
Grateful to the organizers and everyone who shared their amazing research! And I’m gonna miss this beautiful city - Bangkok!
Happy PI🥧 day!
Can language agents🤖 learn social skills through imitation and interaction?
We are excited to introduce SOTOPIA-π🥧 , an interactive learning method for training language agents to navigate real-world social scenarios while role-playing!
Happy PI🥧 day!
Can language agents🤖 learn social skills through imitation and interaction?
We are excited to introduce SOTOPIA-π🥧 , an interactive learning method for training language agents to navigate real-world social scenarios while role-playing!
It also uncovers an interesting challenge! 🤔LLM-based eval tends to overestimate the abilities of language agents specifically trained for social interaction.
Time to think about more advanced evaluation methods to assess social intelligence! ⚖️
Ever used an LLM to synthesize data & wondered about its hallucinations?
Or thought about giving your AI an authorship credit for co-writing a paper?
Join us in uncovering the ethical territories of LLMs & shape a better digital tomorrow! 🤖
✍️Survey:
If you are at
#MSLD2024
/UMich, visit our poster No. 30 today at 11:30 and talk tomorrow morning at 9:45. Both presented by awesome
@RuiyiWang153
.
She will be introducing our newest method for training socially intelligent agents. Go talk to her about anything related to LLMs,
(7/n) Auto-eval⚙️ reveals:
LLM evaluators (GPT-4, Llama 3 70B) underestimate Patient-Ψ🤖’s fidelity compared to GPT-4 baseline. However, GPT-4's accuracy evaluation aligns closely with expert eval.
LLMs excel at understanding cognitive models but fall short in assessing realism
SOTOPIA-π🥧 combines behavior cloning (BC) and self-reinforcement (SR) training on filtered social interaction data, all rated by GPT-4.
The result? 💥A 7B LLM (Mistral-7B-Instruct) that reaches the social goal completion ability of a GPT-4 agent under automatic evaluation!🎉
(4/n) User study shows:
Compared with the GPT-4 baseline, Patient-Ψ🤖offers *more realistic* patient simulations in fidelity dimensions of maladaptive cognitions, emotional states, and conversational styles.
Patient-Ψ-Trainer🛠️is *more effective* for training practitioners.
(5/n) User study also shows:
Experts rated Patient-Ψ🤖 as *very accurate* on average in reflecting the reference cognitive models used to create the simulated patient.
Surprising finding! 💥 SOTOPIA-π🥧 not only improves the agent’s social intelligence, but also enhances safety & reduces toxicity🛡️.
SOTOPIA-π🥧 has the potential to create more socially aligned language agents😊.
(3/n) Key components of building Patient-Ψ🤖:
1️⃣Using cognitive conceptualization diagrams in CBT as cognitive models
2️⃣Creating the first dataset of cognitive models grounded in CBT
3️⃣Integrating six conversational styles
4️⃣Real-time feedback for trainees
🔍How does SOTOPIA-π🥧 work?
1⃣Generate diverse social tasks by sampling keywords from datasets and prompting GPT-4
2⃣Collect training data through interactions between GPT-4 (BC) and the agent being trained (SR)
3⃣Update the agent’s policy using GPT-4 rated positive examples
(2/n) We propose Patient-Ψ-Trainer🛠️, an interactive framework for practicing CBT skills on Patient-Ψ🤖.
A user study of 33 mental health experts and trainees shows Patient-Ψ🤖 has high fidelity to real patients and Patient-Ψ-Trainer🛠️ is highly effective for training.
(6/n) Additionally, for Patient-Ψ-Trainer🛠️:
1️⃣Experts think it can improve trainees’ confidence over the baseline and traditional methods.
2️⃣Experts find value in the real-time feedback
3️⃣Expert prefer integrated conversational styles.
Moreover, SOTOPIA-π🥧 maintains the agent’s general question-answering ability, as demonstrated by its performance on MMLU tasks. 🧠
This shows that SOTOPIA-π🥧 training doesn’t compromise the model’s overall capabilities. ✅
We evaluate SOTOPIA-π🥧 by simulating interactions between our agent & a GPT-3.5 partner, then assess using GPT-4 & human ratings📊.
The evaluation is based on the SOTOPIA framework (explore more on ).