Yijia Shao Profile Banner
Yijia Shao Profile
Yijia Shao

@EchoShao8899

2,357
Followers
296
Following
11
Media
146
Statuses

CS Ph.D. student @StanfordNLP . Previous: undergraduate @PKU1898 .

Joined March 2022
Don't wanna be here? Send us removal request.
Pinned Tweet
@EchoShao8899
Yijia Shao
9 days
There is an illusion that adding humans to the loop is a free lunch for improving AI results or UX. When trying to make STORM collaborative, I am convinced this is NOT true! Check out @YuchengJiang0 's thread to see how we 𝘀𝗰𝗮𝗳𝗳𝗼𝗹𝗱 𝗵𝘂𝗺𝗮𝗻 𝗽𝗮𝗿𝘁𝗶𝗰𝗶𝗽𝗮𝘁𝗶𝗼𝗻⬇️
@YuchengJiang0
Yucheng Jiang
9 days
Struggling to ask the right questions on complex topics? Overwhelmed by too much information? Need a personalized, in-depth report? 🚀 Introducing Co-STORM, a multi-agent system enabling human participation in agent conversations for complex information seeking and learning.
8
46
245
1
11
59
@EchoShao8899
Yijia Shao
6 months
Can we teach LLMs to write long articles from scratch, grounded in trustworthy sources? Do Wikipedia editors think this can assist them? 📣Announcing STORM, a system that writes Wikipedia-like articles based on Internet search. I now use STORM in my daily research!🧵
41
202
1K
@EchoShao8899
Yijia Shao
1 year
New blog post sharing an international student's perspective on applying for CS/AI/NLP PhD in the US! After an intense week of settling into a new country and meeting new people @stanfordnlp , finally got some time to write this post. Hope it may help!
8
41
243
@EchoShao8899
Yijia Shao
5 months
Thrilled to announce we’ve received IRB approval to launch our web demo of STORM at ! 🌪️ While we’ve analyzed its limitations in our paper, we’re eager to kick off a real-world exploration. Try it out, and give us your feedback directly through the demo!
@EchoShao8899
Yijia Shao
6 months
Can we teach LLMs to write long articles from scratch, grounded in trustworthy sources? Do Wikipedia editors think this can assist them? 📣Announcing STORM, a system that writes Wikipedia-like articles based on Internet search. I now use STORM in my daily research!🧵
41
202
1K
7
31
196
@EchoShao8899
Yijia Shao
4 months
Thanks for implementing our paper! But actually, you only need to modify 5 lines of code to configure STORM with Claude models. ZERO line of change is needed now because I just added an example script to our repo!
@alexalbert__
Alex Albert
4 months
STORM by @angelina_magr @MehdiAllahyari Implementation of the paper STORM (Synthesis of Topic Outlines through Retrieval and Multi-perspective Question Asking) -- uses Claude + sub-agents to write long-form articles.
Tweet media one
4
12
125
4
19
142
@EchoShao8899
Yijia Shao
5 months
Since launching STORM code & web preview, thousands have tried it & offered feedback. - Can I run STORM with open LMs? - Can I change its report style? - Can I contribute to new info source support? Yes! We refactored our codebase for smoother running, customization & dev! 🔗🧵
Tweet media one
6
16
104
@EchoShao8899
Yijia Shao
6 months
Check out the Mistral training script at @ericzelikman 's repo ! The model checkpoint can now be downloaded from huggingface 🎉
@ericzelikman
Eric Zelikman
6 months
Language models today are trained to reason either 1) generally, imitating online reasoning data or 2) narrowly, self-teaching on their own solutions to specific tasks Can LMs teach themselves to reason generally?🌟Introducing Quiet-STaR, self-teaching via internal monologue!🧵
35
187
1K
1
10
71
@EchoShao8899
Yijia Shao
2 months
It’s really amazing to grow with the STORM project! To me, the coolest thing is as the system gets more robust and with more people trying it, we get a chance to explore something we cannot do before. You probably notice a hidden mode on the UI - yes, more updates come soon!
@YuchengJiang0
Yucheng Jiang
2 months
🚀 Exciting updates in STORM! 🌟 Research preview 2.0 with improved stability and UX) 📃 Support customized sources 💻 Release a light UI for developers After STORM, we are exploring human-AI collaboration on knowledge tasks. More to come in this summer. Stay tuned!
Tweet media one
10
29
155
1
13
67
@EchoShao8899
Yijia Shao
3 months
My first in-person conference at #NAACL2024 🇲🇽 @_Yucheng_Jiang and I will present STORM🌪️ 𝘁𝗼𝗱𝗮𝘆 𝗮𝘁 𝟮𝗣𝗠 (📍Don Diego 2,3,4).  Would love to chat about report generation, lessons learned in hosting the web demo, and share a bit of what we are cooking after STORM!
@EchoShao8899
Yijia Shao
6 months
Can we teach LLMs to write long articles from scratch, grounded in trustworthy sources? Do Wikipedia editors think this can assist them? 📣Announcing STORM, a system that writes Wikipedia-like articles based on Internet search. I now use STORM in my daily research!🧵
41
202
1K
2
8
64
@EchoShao8899
Yijia Shao
6 months
Generating long articles with citations is hard to do & hard to evaluate! We break this problem down into two steps: 1️⃣Pre-writing, in which the system collects references and generates an outline. 2️⃣Writing, in which the system generates the final article with citations.
Tweet media one
1
7
58
@EchoShao8899
Yijia Shao
9 months
Continual learning in NLP!?! Tried to explore this with language models during my undergraduate and my bachelor's thesis is exactly about this topic (, ). What shall we do when we shift to LARGE language models?
@annargrs
Anna Rogers
9 months
@chrmanning at #EMNLP2023 to #NLProc PhD students, who are having an existential crisis over LLMs: Aeronautics students do not build Boeings for their PhD theses. They do smaller models - and still make meaningful contributions. There's plenty of such opportunities for us too.
Tweet media one
Tweet media two
5
76
393
1
5
54
@EchoShao8899
Yijia Shao
4 months
Check out @kenziyuliu 's blog post about machine unlearning! It's a very thoughtful overview along with cool insights on new research directions. Also, I was greatly impressed by how he did literature review and organized ideas when I first read the draft :)
@kenziyuliu
Ken Liu
4 months
The idea of "machine unlearning" is getting attention lately. Been thinking a lot about it recently and decided to write a long post: 📰 Unlearning is no longer just about privacy and right-to-be-forgotten since foundation models. I hope to give a gentle
23
162
745
1
4
49
@EchoShao8899
Yijia Shao
7 days
Start to get questions about grad school application this week. Repost a blog I wrote last year, sharing experience from an international student’s perspective 😎
@EchoShao8899
Yijia Shao
11 months
Just moved my website. Here is the new link to this post.
0
0
9
4
3
50
@EchoShao8899
Yijia Shao
6 months
@laion_ai We will open source the code for sure.
4
0
43
@EchoShao8899
Yijia Shao
6 months
“Pre-writing” requires researching a topic from scratch. That makes it hard even for expert humans. And directly prompting the LM to generate questions doesn’t work well! The questions lack depth and have limited breadth. STORM is designed to teach LMs to *ask good questions*.
Tweet media one
1
4
41
@EchoShao8899
Yijia Shao
6 months
It’s worth mentioning that STORM is a carefully designed pipeline for knowledge curation rather than a single prompt or model. We build STORM using DSPy which provides very neat modularization - this allows us to keep extending our work without getting lost in many prompt files.
2
1
32
@EchoShao8899
Yijia Shao
6 months
In the final writing stage, STORM generates text with citations and writes the full article section by section. Articles produced by STORM are favored by both automatic metrics *and* experienced Wikipedia editors!
Tweet media one
1
3
28
@EchoShao8899
Yijia Shao
6 months
STORM improves question asking by automatically discovering perspectives for researching the topic and adding the perspective in the prompt. It also simulates information-seeking conversations to encourage follow-up questions which are usually more in-depth.
Tweet media one
1
3
27
@EchoShao8899
Yijia Shao
6 months
We build FreshWiki to mitigate data leakage into LM training data for evaluation. To measure quality, we introduce heading soft recall and heading entity recall. Outline eval makes it easier to prototype methods for pre-writing. STORM outperforms well-designed RAG baselines!
Tweet media one
1
4
25
@EchoShao8899
Yijia Shao
6 months
Such expository writing should always be grounded. We assess citation quality and ask Wikipedia editors to rate verifiability. We find the major challenge stems from red herring rather than widely discussed factual hallucination. This calls for research beyond fact-checking!
Tweet media one
1
0
24
@EchoShao8899
Yijia Shao
9 months
Like this idea, though unsure about what's the right analogy, as LLMs process text with uncertainty. Tried to use the analogy of OS in a discussion with people in the mathematics department @PKU1898 (), but feel the proactive power of LLMs is a bit missing.
@lateinteraction
Omar Khattab
9 months
Actionable, extended analogies are underrated. Here's one: LMs are stochastic *devices* for pattern processing. Like CPUs or GPUs, but Language Processing Units (LPUs). Assembly? Higher-Level Languages? Compilers? Understanding this analogy resolves several major questions.🧵
5
25
110
1
2
18
@EchoShao8899
Yijia Shao
6 months
We also ask Wikipedia editors for the perceived usefulness of STORM. It’s exciting that all participants agree that STORM is helpful for their pre-writing stage. Also, I use STORM myself to learn concepts in-depth in my research 😎(check out our demo video if you haven’t).
Tweet media one
1
0
16
@EchoShao8899
Yijia Shao
7 months
Check out our #ICLR2024 paper that achieves negligible forgetting in continual learning (CL)! CL ability is crucial in the ever-evolving open world. Another crucial problem in open-world ML is out-of-distribution detection. This work links these two and shows remarkable results!
@AndyLin2001
Haowei Lin
7 months
Solving continual learning via our likelihood-ratio based task prediction method can achieve negligible forgetting🧠! Checkout details🧐in this thread and our ICLR 2024 paper. Paper📄: Code🔧:
2
0
4
1
1
14
@EchoShao8899
Yijia Shao
5 months
The official implementation of STORM is now released at . We are working on enhancing our codebase to increase its extensibility. Contributions and suggestions are welcome!
1
2
14
@EchoShao8899
Yijia Shao
1 year
What's special when using PLMs for continual learning? Our #ACL2023 paper shows PLM's class-incremental learning performance is related to its representation ability in the learning process and the generation objective helps a lot. #NLProc 🔗: !
Tweet media one
1
1
14
@EchoShao8899
Yijia Shao
6 months
@helloiamleonie @lateinteraction Thanks for sharing our work!! We hope STORM can be an assisting tool that reduces laborious searches in the pre-writing stage and mitigates echo chamber in information seeking. We are working on letting more people try it and interested to know what people may use it for.
3
0
13
@EchoShao8899
Yijia Shao
9 months
Usually do random walk in paper skimming in the morning and spotting this one leads to an extra long breakfast😂 Sparsity is a word people like but hard to make it work or even efficiently implement. The paper’s motivation part is very inspiring!
@srush_nlp
Sasha Rush
9 months
Goddamn, wild times. Here's the paper you need to read understand today: have fun.
Tweet media one
13
132
1K
1
0
12
@EchoShao8899
Yijia Shao
8 months
Super admired this effort after listening to an internal talk last week! Reflecting on my personal experience before college, I think good tutoring is still very limited. This line of work really provides me with a concrete example of NLP for social good.
@rose_e_wang
Rose
8 months
There’s a *lot* of insight about how students learn in education data like classroom conversations, but I can tell you it's *painful* to process and analyze. Introducing *Edu-ConvoKit*, a pipeline that handles the pre-processing, annotation, and analysis for you! 🧵
Tweet media one
2
19
99
1
2
10
@EchoShao8899
Yijia Shao
2 months
Sad that I cannot go due to visa issue. But my amazing collaborator @_Yucheng_Jiang will give the pre in person. He will talk about STORM and a bit about new stuff. It’ll be a good chance for communication! The event happens in SF on July 11. Detailed event info in his post.
@YuchengJiang0
Yucheng Jiang
2 months
We’ll be giving an invited talk at the Wikipedians Meetup @Wikipedia @Wikimedia in San Francisco on July 11. The event happens at 6.30 PM - 8.30 PM.  Come to chat with us in person to learn more!
1
1
9
0
0
10
@EchoShao8899
Yijia Shao
6 months
@realmrfakename We will open-source the code for sure. We are also working on making the demo available as we found it provides a more straightforward way to try STORM engine in our internal test.
0
0
10
@EchoShao8899
Yijia Shao
2 years
#EMNLP2022 Creating online forms has become one of the tiring chores these days. Check our FormLM paper () to see how AI can help! This work is done @MSFTResearch and I will present it at 15:30 local time today at EMNLP Virtual Poster Session 10!😉
Tweet media one
3
1
10
@EchoShao8899
Yijia Shao
6 months
Check out @wangyenjen 's work on humanoid robot training framework! Heard Yen-jen talk about the possibility of using LLM for high-level planning when ChatGPT just came out. It's so cool to see how it handles various challenging situations!
@roboterax
ROBOTERA
6 months
🚀RobotEra & @Tsinghua_Uni launch Humanoid-Gym! An open-source sim2real RL framework for humanoid robots. This codebase is verified on RobotEra's humanoid robot XBot (XiaoXing)! Project page: Github : @JianyuChen_THU @wangyenjen
4
38
120
1
2
8
@EchoShao8899
Yijia Shao
5 months
Huge shout out to @_Yucheng_Jiang ! And we are very grateful to @YouSearchEngine for the generous support.
0
0
7
@EchoShao8899
Yijia Shao
9 days
@YuchengJiang0 Besides people on the author list, we are super grateful to @lateinteraction , @ericzelikman , @rose_e_wang , @wangyenjen , and Qingyang Tang for helping us at different stages of this work.
1
0
7
@EchoShao8899
Yijia Shao
9 months
Task-incremental learning may not be necessary as there is a meta-task called instruction following. But continual learning is still important as we may envision a model/system that can continually learn to better serve specific users or become better in challenging domains.
1
0
5
@EchoShao8899
Yijia Shao
9 months
Speaking of this, I am interested in building models/systems with initiatives - question answering is important but just passively answering questions is not enough.
1
0
6
@EchoShao8899
Yijia Shao
6 months
@quentinferry1 For results in the paper, we use @YouSearchEngine search API (huge thanks to them for supporting my research!). We use DSPy by @lateinteraction to build the whole system because it's a lightweight framework. I found it also extensible when doing frontend integration for our demo.
0
0
6
@EchoShao8899
Yijia Shao
9 days
@YuchengJiang0 @lateinteraction @ericzelikman @rose_e_wang @wangyenjen Also, great shout out to our human evaluation participants and all contributors to . It's surreal seeing 10k stars and getting many external PRs. We are working to integrate co-storm into the knowledge-storm package. Stay tuned!
0
0
6
@EchoShao8899
Yijia Shao
16 days
@WikiResearch Thanks for covering our work in a careful and objective way!!
0
0
4
@EchoShao8899
Yijia Shao
5 months
Thanks for sharing! Yes, llama3 is worth a try. Also, it'll be interesting to see whether @ollama integration is possible as people are discussing this in our GitHub repo.
@lateinteraction
Omar Khattab
5 months
STORM embodies many of the best design choices for Language Programs—and is probably the most popular open-source repo written in DSPy. Check it out! @EchoShao8899 & @_Yucheng_Jiang just added support for local models and other custom choices. I bet it’s worth a try with Llama3.
1
20
96
1
1
5
@EchoShao8899
Yijia Shao
2 months
Definitely worth chatting with @wangyenjen to learn about how to make robots work 🤖🦾🦿
@wangyenjen
Yen-Jen Wang
2 months
#RSS2024 We will present our DWL in the main session 9. Locomotion and manipulation (Wednesday). If you're interested, don't hesitate to chat with @JianyuChen_THU and me. Looking forward to meet all of you in Delft! : ) Advancing Humanoid Locomotion: Mastering Challenging
Tweet media one
0
1
28
0
1
5
@EchoShao8899
Yijia Shao
15 days
Amazing reading group with amazing support and an amazing organizer!🤩
@krisgligoric
Kristina Gligorić
15 days
Grateful to @StanfordHAI for supporting our group. Check out the interview to learn more about it!
0
10
111
0
0
4
@EchoShao8899
Yijia Shao
5 months
Thank @_Yucheng_Jiang for the amazing collaboration to make things happen!
1
0
4
@EchoShao8899
Yijia Shao
6 months
@patrickswnsn Thank you so much, Patrick! Really nice to have you guys as our alpha users and iterate upon your feedback!!
0
0
4
@EchoShao8899
Yijia Shao
11 months
Just spot this brilliant work! I personally like the OS analogy (happened to share this analogy early this year ). But what bothers me is how to discover/recover errors when using LLMs to manage control flow. Any better way than adding tons of try-except?
@charlespacker
Charles Packer
11 months
Introducing MemGPT 📚🦙 a method for extending LLM context windows. Inspired by OS mem management, it provides an infinite virtualized context for fixed-context LLMs. Enables perpetual chatbots & large doc QA. 🧵1/n Paper: GitHub:
10
105
461
1
1
4
@EchoShao8899
Yijia Shao
6 months
@Moi39017963 We have not tried debate - it is a good suggestion! But one reason that we do not have interaction between different perspectives is that in this design, they can run in parallel. For self-critique, we have tried adding one refinement step but later remove it for simplicity.
1
0
2
@EchoShao8899
Yijia Shao
1 year
@_ddjohnson @dan_fried A very good inspiring perspective, thanks for sharing it!! I totally agree that we should take uncertainty/confidence into account when "sampling texts" from LLMs. Are there any useful metrics to track the model's confidence in its own output?
1
0
3
@EchoShao8899
Yijia Shao
8 days
@rose_e_wang @YuchengJiang0 Thanks, Rose!! A large part of the discussion goes into how we structured the method section. Also, I was impressed by how knowledgeable you are as I didn’t expect when chatting, you came up with different concepts & pointers on notebook instead of search engine😀
0
0
3
@EchoShao8899
Yijia Shao
7 months
This is my last work that started at PKU with Prof. Liu's group. We have been interested in building models that can keep learning in the changing world. I still remember that my first research project was designing an OOD detection algorithm with @AndyLin2001 .
1
0
3
@EchoShao8899
Yijia Shao
7 months
@AndyLin2001 After that, we studied CL, focusing on continual pre-training/fine-tuning for language models. Finally, this work led by @AndyLin2001 connects OOD detection and CL. It's amazing to collaborate with Haowei on all these projects - it makes me learn a lot and have a lot more fun.
0
0
3
@EchoShao8899
Yijia Shao
2 months
Super grateful to @_Yucheng_Jiang and Dekun Ma from Yale for working together on this, to @lateinteraction and @MonicaSLam for all helpful advice. We would also like to thank @Microsoft Azure and @yousearchengine for the generous support!
0
0
2
@EchoShao8899
Yijia Shao
5 months
@_Yucheng_Jiang Also, huge thanks to @Azure , @NSF , @SloanFoundation , @StanfordHAI , @YouSearchEngine for supporting this research project!
0
0
2
@EchoShao8899
Yijia Shao
3 months
@mysticaltech STORM is already open-sourced at . We plan to release a few features that people have been asking for (e.g., UI for developers) when we return, so follow our GitHub repo!
1
0
2
@EchoShao8899
Yijia Shao
1 year
@JingfengY Thank you for sharing this!!
0
0
2
@EchoShao8899
Yijia Shao
4 months
@VioletNPeng Congratulations, Nanyun!!!
0
0
2
@EchoShao8899
Yijia Shao
2 years
@JingfengY Thanks for sharing! Just found some time to read this and I am wondering whether expecting a single model to achieve everything is the correct thing...
1
0
1
@EchoShao8899
Yijia Shao
3 months
@johnhewtt Congratulations!!
0
0
1
@EchoShao8899
Yijia Shao
2 months
@dorazhao9 congrats!🎊
0
0
1
@EchoShao8899
Yijia Shao
1 year
@wangyenjen Thank you for sharing this!!
0
0
1
@EchoShao8899
Yijia Shao
9 months
@ruoerdi Yeah, given the fact that we have instruction-following as a meta-task and can use newly collected facts through retrieval, I feel it's also important to think about what data source to continually learn from besides designing better algorithms.
1
0
1
@EchoShao8899
Yijia Shao
5 months
@julianharris Thanks! Could you retry it? We got high-volume access this morning and just addressed several issues.
1
0
1
@EchoShao8899
Yijia Shao
8 months
0
0
1
@EchoShao8899
Yijia Shao
6 months
@ericzelikman @xai @Stanford Awesome! Congratulations, Eric!!
0
0
1
@EchoShao8899
Yijia Shao
11 months
LLMs' non-deterministic nature sometimes bother me when I try to use them for the controller purpose.
0
0
1
@EchoShao8899
Yijia Shao
5 months
@_0xcore Thanks! Currently, you can follow our Github repo . We will have follow-up release to make the curation engine more extensible as we have feedback hoping to use STORM for different use cases. Looking forward to your contribution!!
0
0
1
@EchoShao8899
Yijia Shao
6 months
@sushant4573 We will open-source and are also working on letting people try through the user interface.
2
0
1
@EchoShao8899
Yijia Shao
4 months
@tombielecki @houshuang @SemanticScholar No, because I haven't found any good tool yet. I think this use case is an interesting human-AI collaboration scenario.
0
0
1
@EchoShao8899
Yijia Shao
1 year
@DeveshVall96058 @stanfordnlp Thank you!! Since I'm new here, I cannot involve others right away. But I would suggest not excluding industry opportunities as they usually have more headcounts. You can select groups that are catching up with new thing to get some experience. That helps the next step!
1
0
1
@EchoShao8899
Yijia Shao
2 years
Welcome to have a chat! I am also applying for a PhD in NLP :) For friends in Mainland China, our paper video is also available on Bilibili ().
0
0
1