Yihe Deng @Yihe__Deng Twitter profile

Pinned Tweet

Yihe Deng

4 months

📢 New paper alert! Introducing STIC (Self-Training on Image Comprehension) that enhances the understanding and reasoning capabilities of LVLMs through self-generated data 🌟 📄 Read the paper: 🔗 Project page: 💻 GitHub Repo:

1

44

198

Last Seen Profiles

@DaniYEduardo2

@TotalPackageAPH

@madamruth26

@gaby_sepulveda

@AgentP22

@thekitchn

@Finkerton

@MonicaSaska99

@KHEinkauf

@himeyuri_peace

@_hadime_

@RockCasbah15620

@onlysixshapes

@JtmoneyG1

@WeAHPs

@MM_newsonline

@TaofeekOgunperi

@tireamisu_

@Sunil_Deodhar

@GranCTimes

@MorpheusResisst

@11chigua

@SilverD_64

@twcityjournal

@Sir__Mike

@imyeol02

@benji_gotfried

@Mtmorency_IDFM

@TDLP17

@ThePissMan2

@GuillermoRoz1

@faye_ttan

@Nezzified

$ellen_{SpEcIaLiSt /aRtiSt /cOmM}$

@Artistellen1

@UHpres

Yihe Deng

@Yihe__Deng

9 months

Thanks @arankomatsuzaki for sharing our work! Introducing Self-Play fIne-tuNing (SPIN) for LLM’s iterative self-improvement. 👇

Aran Komatsuzaki

@arankomatsuzaki

9 months

Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models Significantly improves the LLM’s performance across a variety of benchmarks and even outperform models trained through DPO with extra GPT-4 preference data

14

63

446

10

42

229

Yihe Deng

@Yihe__Deng

8 months

Large Vision Language Models are prone to object hallucinations – how to cost-efficiently address this issue? 🚀 Introducing MARINE: a training-free, API-free framework to tackle object hallucinations. Joint work with an amazing team @linxizhao4 @WeitongZhang and @QuanquanGu !

4

46

172

Yihe Deng

@Yihe__Deng

1 year

(1/2) Ever wondered why CLIP shines in zero-shot transfer?🌟Our new arXiv paper dives into the theoretical mechanism behind it. We explain how CLIP's contrastive objective enables transferrable representation learning in multi-modal data. 🔍📄 arXiv:

1

13

104

Yihe Deng

@Yihe__Deng

11 months

Codes and project page are released for our #NeurIPS23 paper on spurious correlations in robust learning! 🚀 🔗 Project: 🔗 Code: Key Insights: 📊 We discovered in theoretical analysis that spurious features overtake initial

GitHub - uclaml/PDE: Official repo of Progressive Data Expansion: data, code and evaluation

Official repo of Progressive Data Expansion: data, code and evaluation - uclaml/PDE

github.com

2

14

90

Yihe Deng

@Yihe__Deng

8 months

Happy year of loong! Small milestones for me this past year🥹 - Two of our papers are accepted to #ICLR2024 ! - We released two arxiv papers on LLMs: RaR and SPIN. Give them a check😄 - ⭐️The code for SPIN is now open-sourced! Dive into it here:

2

1

69

Yihe Deng

@Yihe__Deng

1 year

Happy to share that our work "Robust Learning with Progressive Data Expansion Against Spurious Correlation" has been accepted to #NeurIPS2023 ! 🎉 arXiv:

3

60

Yihe Deng

@Yihe__Deng

10 months

1st day at #NeurIPS2023 Can’t wait to learn more about LLMs 😄 Would love to chat about research!

0

6

48

Yihe Deng

@Yihe__Deng

3 months

🚀Check out our new benchmark paper on LLM agents for global events forecasting! More interesting examples in our project page and colab demo 👇

Chenchen Ye

@chenchenye_ccye

3 months

📢New LLM Agents Benchmark! Introducing 🌟MIRAI🌟: A groundbreaking benchmark crafted for evaluating LLM agents in temporal forecasting of international events with tool use and complex reasoning! 📜 Arxiv: 🔗 Project page: 🧵1/N

14

71

305

0

3

34

Yihe Deng

@Yihe__Deng

10 months

I’m here! 🤣🤣🤣 Fantastic time chatting with everyone!

Yann LeCun

@ylecun

10 months

Meta party at #neurips2023 Giant selfie.

21

16

599

0

1

33

Yihe Deng

@Yihe__Deng

1 year

📢Excited to be attending #icml2023 soon and thrilled to meet and connect with everyone! I'll be presenting at SCIS workshop @ Sat 29th 2:30-3:30pm Meeting Room 316 AB for our work "Robust Learning with Progressive Data Expansion Against Spurious Correlation."

2

3

31

Yihe Deng

@Yihe__Deng

11 months

Amazing new feature! Just created RaR-GPT that will rephrase the question first before responding. 😉Give it a try!

Sam Altman

@sama

11 months

GPTs are now live for all ChatGPT+ subscribers!

766

1K

15K

0

2

23

Yihe Deng

@Yihe__Deng

10 months

OPT poster session 1 at Hall D2 👇

Xuheng Li

@xuhengli_

10 months

Curious about when accelerated SGD (ASGD) outperforms SGD? Our latest work provides new insights about the comparison between ASGD and SGD in overparameterized linear regression. (1/3) 🔗 Joint work with @Yihe__Deng @uuujingfeng @DongruoZ and @QuanquanGu

3

6

37

1

2

21

Yihe Deng

@Yihe__Deng

1 year

😄Enjoying the great vibes at #ICML2023 ! Epic to meet all the Amazonians here and connect with the #AmazonScience crew. 🤙

1

2

20

Yihe Deng

@Yihe__Deng

11 months

Happy see to that our method RaR is working for ChatGPT as well! 😄 Check more on our project page for our discoveries and interesting examples:

Rene Heuser

@reneheuser

11 months

@QuanquanGu Thanks for sharing. It’s a nice short Custom Instruction for ChatGPT, too. With this CI it correctly created the „10 sentences that ends with Apple test“ … and of course the examples from your paper.

0

4

0

19

Yihe Deng

@Yihe__Deng

10 months

I'll be at #707 from 5:15 — 7:15 pm. Drop by and chat with me! 😄

Yu Yang

@YUYANG_UCLA

10 months

Don't miss our poster session today at #NeurIPS2023 ! 🤗 @Yihe__Deng will be presenting our work on "Robust Learning with Progressive Data Expansion Against Spurious Correlation." 📍 Great Hall & Hall B1+B2 (level 1) #707 ⏰ 5:15 p.m. - 7:15 p.m. CST 🔗

0

7

65

0

15

Yihe Deng

@Yihe__Deng

1 year

Check out this great work on benign overfitting in two-layer ReLU CNNs at #ICML2023 🙌by @YiwenKou666 , @_zxchen_ , Yuanzhou and @QuanquanGu ! 📍 Exhibit Hall 1, Poster #603 📅 Thurs, 27th July (Today!), 10:30 a.m. - 12:00 p.m. HST

Zixiang Chen

@_zxchen_

1 year

Join us at ICML2023 for insights on "Benign Overfitting in Two-layer ReLU Convolutional Neural Networks" Our research reveals a sharp transition between benign and harmful overfitting in two-layer ReLU CNNs with label-flipping noise. (1/3)

1

3

0

12

Yihe Deng

@Yihe__Deng

10 months

@furongh I’m here as well. The line is insane 😂

1

0

9

Yihe Deng

@Yihe__Deng

1 year

(2/2) We also analyze the effect of temperature parameter in CLIP on representation learning.🌡️Our theory suggests a regularization term that provably improves CLIP's original objective. Joint work with @_zxchen_ , Yuanzhi Li and @QuanquanGu 👏

0

9

Yihe Deng

@Yihe__Deng

8 months

@linxizhao4 @WeitongZhang @QuanquanGu 3/ 📈 POPE: the POPE metric similarly validates the superior performance of MARINE against existing baselines on different question formats, achieving improvements of up to +21.4% on accuracy and +12.0% on F1 score. Refer to more experiment details and examples in our paper!

0

1

7

Yihe Deng

@Yihe__Deng

10 months

@JasonForJoy @furongh Optimizing and reasoning about LLM inference :D

0

4

Yihe Deng

@Yihe__Deng

8 months

@linxizhao4 @WeitongZhang @QuanquanGu 1/ 🎯 Our Study: MARINE vs. Baselines. We used CHAIR and POPE metrics, focusing not just on initial LVLM versions (like LLaVA) but also on advanced ones (e.g., LLaVA-v1.5). Overall, MARINE achieves superior performances across different LVLM architectures and evaluation metrics,

1

0

5

Yihe Deng

@Yihe__Deng

8 months

@linxizhao4 @WeitongZhang @QuanquanGu Current mitigation methods, while becoming more cost-efficient, still rely on fine-tuning models or using GPT-3.5 for post-generation corrections. Effective? Yes, but they tend to overwrite original model outputs, losing inherent diversity and instruction adherence. See this

1

5

Yihe Deng

@Yihe__Deng

1 year

👏Check out this amazing work on interplay between misspecification with the sub-optimality gap in linear bandits by my labmate @WeitongZhang and team, to be presented at #ICML2023 ! 👇

Weitong Zhang

@WeitongZhang

1 year

At #ICML2023 : How will the function approximation error affect the performance of online sequential decision making? Find us in Poster Session 1 @ Exhibit Hall 1 #627 from 11 a.m. HST to 1:30 p.m. HST. Looking forward to discussing with you all! (1/3)

1

9

0

5

Yihe Deng

@Yihe__Deng

2 months

@YUYANG_UCLA @VirtueAI_co Congrats and best wishes 🎉

1

0

4

Yihe Deng

@Yihe__Deng

4 months

We conduct extensive experiments on seven vision-lanugage benchmarks, encompassing scientific reasoning, math reasoning, OCR, and conversation capabilities based on vision inputs, spanning image sources such as natural, chart, and text-rich images. With LLaVA-v1.6 as our primary

1

0

4

Yihe Deng

@Yihe__Deng

8 months

@linxizhao4 @WeitongZhang @QuanquanGu 1/ 🌟 MARINE's Core: it's all about combining object grounding features with CFG to control text generation, aiming to reduce object hallucinations in LVLMs. 2/ 🤖 How it works: we integrate visual features from an object grounding encoder. Using 'direct alignment', we map

1

0

4

Yihe Deng

@Yihe__Deng

4 months

Details for the self-constructed preference dataset: Based on a collection of unlabeled images, we focus the image description task and let the model self-construct a preference dataset via the following: 1. Preferred response is generated from a “good” prompt, which is

1

0

3

Yihe Deng

@Yihe__Deng

8 months

@ChujieZheng @QuanquanGu Yes, we use a global batch size of 64 and require 8*A100 80G machine.

0

3

Yihe Deng

@Yihe__Deng

9 months

@winglian @arankomatsuzaki Thank you for the comment. Our codebase is based on the Alignment Handbook library by HuggingFace, and we're planning to release our code in the near future. Stay tuned for updates!

1

0

3

Yihe Deng

@Yihe__Deng

4 months

Outline of our method: 1️⃣ Stage 1: The model self-constructs a preference dataset using unlabeled images, generating high-quality image descriptions through a detailed “good” prompt and generating less accurate responses via corrupted images or misleading “bad” prompts. 2️⃣

1

0

4

Yihe Deng

@Yihe__Deng

4 months

Specific examples show that STIC enables the model to understand the given image first before reasoning with the input query! Find more examples and ablation studies in our paper: [5/N]

0

3

Yihe Deng

@Yihe__Deng

6 months

@uclanlp Thanks for inviting me!😄

0

2

Yihe Deng

@Yihe__Deng

9 months

@convexstrictly @arankomatsuzaki Hi, thanks for your interest! We listed our key takeaways in more detail here, feel free to check it out 😄

Zixiang Chen

@_zxchen_

9 months

Excited to share our method called 𝐒𝐞𝐥𝐟-𝐏𝐥𝐚𝐲 𝐟𝐈𝐧𝐞-𝐭𝐮𝐍𝐢𝐧𝐠 (SPIN)! 🌟Without acquiring additional human-annotated data, a supervised fine-tuned LLM can get stronger by SPIN. Check out how SPIN unleashes the full power of human-annotated data. Joint work with

12

75

326

0

2

Yihe Deng

@Yihe__Deng

9 months

@lvwerra Thank you! Will keep in touch once we release models/codes😄

0

1

Yihe Deng

@Yihe__Deng

8 months

@Leon_L_S_C @linxizhao4 @WeitongZhang @QuanquanGu Hi, thanks for mentioning the related works! We did compare with the reported results of VCD on LLaVA-v1.5 and InstructBLIP using POPE (in our table 2).

0

2

Yihe Deng

@Yihe__Deng

9 months

@Yuancheng_Xu0 @arankomatsuzaki Thanks! We observed is that even after fine-tuning the LLM on a given SFT dataset, there is still a noticeable quality gap between its response to a training prompt and the ground truth completion. At high level, SPIN is aimed to let the LLM itself discern the gap and

0

2

Yihe Deng

@Yihe__Deng

9 months

@ChujieZheng @iclr_conf Congrats!!

1

0

1

Yihe Deng

@Yihe__Deng

9 months

@arnicas @arankomatsuzaki Hi and thanks for your interest. Our codebase is based on the Alignment Handbook library by HuggingFace, and we're planning to release our code in the near future. Stay tuned for updates!

0

1

Yihe Deng

@Yihe__Deng

8 months

@ChujieZheng Thanks! Your recent works like DRO are also super interesting and I learned a lot from 🫡

0

1

Yihe Deng

@Yihe__Deng

8 months

🤯

OpenAI

@OpenAI

8 months

Introducing Sora, our text-to-video model. Sora can create videos of up to 60 seconds featuring highly detailed scenes, complex camera motion, and multiple characters with vibrant emotions. Prompt: “Beautiful, snowy

10K

32K

138K

0

1

Yihe Deng

@Yihe__Deng

5 months

@AlbalakAlon @ucsbNLP @ucsantabarbara @synth_labs Congrats Alon! 🎉

1

0

1

Yihe Deng

@Yihe__Deng

8 months

@HaoqinT Sounds very interesting! Will look into this paper 🤔️ thanks for bringing it up

0

1

Yihe Deng

@Yihe__Deng

5 months

@peizNLP Congrats Dr. Zhou!!! 🎉🎉🎉

1

0

1

Yihe Deng

@Yihe__Deng

9 months

@EasonZeng623 Nice work!

0

1

Yihe Deng

@Yihe__Deng

1 year

@QuanquanGu Thank you very much!😄

0

1

Yihe Deng

@Yihe__Deng

11 months

@Thewimo @QuanquanGu @abacaj Hi, thanks for your interest! We indeed tested on Vicuna-13b-v1.5, which is based on Llama-2, and observed improvements using RaR. ()

lmsys/vicuna-13b-v1.5 · Hugging Face

huggingface.co

1

0

1

Yihe Deng

@Yihe__Deng

1 year

We build upon theoretical insights and propose a new training algorithm to efficiently improve robustness against spurious correlations in deep learning models. Looking forward to the insightful discussions and valuable networking opportunities at ICML!

0

1