Chenchen Ye @chenchenye_ccye Twitter profile

Pinned Tweet

Chenchen Ye

1 month

📢New LLM Agents Benchmark! Introducing 🌟MIRAI🌟: A groundbreaking benchmark crafted for evaluating LLM agents in temporal forecasting of international events with tool use and complex reasoning! 📜 Arxiv: 🔗 Project page: 🧵1/N

14

71

299

Last Seen Profiles

@wangjidvthauls

@ray_nythin

@kaelkardashh

@pismo_io

@passunda82

@DensonHill

@RuralSoftball

@MAF_exe

@meyer_lucas

@tineforSB19

@bonottoguerra

@ChloeLuvv2

@StudD

@UNA_Tz

@NorliaNorlia2

@kosunghoo

@Sumor73283685

@stw_pdg

@beatnim

@rishmody

@DusklightGames

@flurumoaopenta

@Averil701

@vitanova1988

@Simmelmanni

@HelmDeVegas

@MIAMoldova

@DWestvirginia

@vla_darkshadow

@MonkInfern61033

@Niko1393347

@ryue_ryu10716

@lilmonix3

@vnxnitta

@cookedbyeli

@67O72

Chenchen Ye

@chenchenye_ccye

1 month

🧵2/N We released our code, data and an iteractive demo: 💻 GitHub Repo: 📁 Dataset: 📊 Interactive Demo Notebook:

1

13

Chenchen Ye

@chenchenye_ccye

1 month

🧵11/N Sincere thanks to all amazing collaborators and advisors @acbuller , @Yihe__Deng , @HuangZi71008374 , @mingyu_ma , @Zhu_Yanqiao , and @WeiWang1973 for their invaluable advice and efforts! 🙏❤️

0

11

Chenchen Ye

@chenchenye_ccye

1 month

🧵 7/N Forecasting with Different Base LLMs 1️⃣ 📈 Code Block benefits stronger LLMs but hurts weaker models. 2️⃣ 🏆GPT-4o consistently outperforms other models. 3️⃣ 💪 Self-consistency makes a small model stronger.

1

0

10

Chenchen Ye

@chenchenye_ccye

1 month

🧵10/N Check our paper out for more details! 🌟 Code error analysis, different event types, variation of API types, and different agent planning strategies! Join us in advancing the capabilities of LLM agents in forecasting and understanding complex international events! 🚀

1

0

10

Chenchen Ye

@chenchenye_ccye

1 month

🧵 4/N Forecasting Task 🔮 Forecasting involves collecting essential historical data and performing temporal reasoning to predict future events. 📅 Example: Forecasting cross-country relations on 2023-11-18 using event and news information up to 2023-11-17.

1

0

10

Chenchen Ye

@chenchenye_ccye

1 month

🧵 8/N Forecasting with Temporal Distance Our ablation study let agents predicts 1, 7, 30, and 90 days ahead. 📊Results: As days increases, F1📉and KL📈. Agent's accuracy drops for distant events. Longer ones anticipate trend shifts influenced by more factors and complexities.

1

0

10

Chenchen Ye

@chenchenye_ccye

1 month

🧵 6/N Agent Framework 💡 Think: Agent analyzes and plans the next action using API specs. ⚡ Act: Generates Single Function or Code Block to retrieve data. 🚀 Execute: Python interpreter runs the code for observations. These steps are repeated until reaching a final forecast.

1

0

10

Chenchen Ye

@chenchenye_ccye

1 month

🧵 9/N Tool-Use Ordering in Forecasting 🗂️Tool-Use Transition Graph: Agents start with recent events for key info and end with news for context. 🧠 Freq.(correct) - Freq.(incorrect): Highlight the need for strategic planning in LLM agents for effective forecasting.

1

0

9

Chenchen Ye

@chenchenye_ccye

1 month

🧵 5/N APIs & Environment 💻 Our comprehensive APIs empower agents to generate code and access the database. 🔧 APIs include data classes and functions for various info types and search conditions. 🔄 Agents can call a single function or generate a code block at each step.

1

0

9

Chenchen Ye

@chenchenye_ccye

1 month

🧵3/N Data 🌐With 59,161 unique events and 296,630 unique news articles, we curate a test set of 705 forecasting query-answer pairs. (a)📊 Circular Chart: The relation hierarchy and distribution in MIRAI. (b-c) 🔥 Heatmap: Intensity of global events, from conflict to mediation.

1

0

9

Chenchen Ye

@chenchenye_ccye

1 month

@nicolayr_ Thanks for sharing your thoughts, Nicolay! Your idea about forecasting from literature sounds really interesting!

1

0

2

Chenchen Ye

@chenchenye_ccye

1 month

@aviaviavi__ Thank you so much, Avi! Looking forward to hearing your thoughts!

0

1