Yilun Xu @xuyilun2 Twitter profile

Last Seen Profiles

@stwgemoy

@Nat_Numeracy

@ixUZ3l0l31f7t

@bokeplokalmalam

@nightynn

@JaylenTyr

@stwmaniax

@gulingtukar

@JACKCleCasino

@ssm_te

@AMRichardson1

@jerkimof

@QasrAlHosn

@mi6_6yu

@theVibesGuru

@maariasm__

@pickleballbp

@ofulover

@vulsya

@sipotexxx

@bokeplokalmalam

@pedapudi007

@niurcmontiel2

@mhmdrezabaratii

@jujuiiuusohot

@hudsonfelker

@bakingcupcake

@nosuke_syousai

@shun66_fn

@stw46

@JuanDeaggg

@laarissaasilva

@sebastiankehl

@markoscales

@tx_analysor

@kimikkk2

Yilun Xu

@xuyilun2

2 years

I curate a list of interesting ICLR submissions on Diffusion/SDE/ODE-based generative models. (). Welcome to add more 🧐

Diffusion/SDE/ODE-based generative models papers (ICLR2023)

Text-to-Image Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesis Unifying Diffusion Models' Latent Space, with Applications to CycleDiffusion and Guidance DreamFu...

docs.google.com

3

83

434

Yilun Xu

@xuyilun2

29 days

Introducing Discrete-Continuous Latent Variable Diffusion Models (DisCo-Diff 🕺), which augment continuous diffusion models with learnable global discrete latents. DisCo-Diff greatly simplify learning diffusion models and strengthens their sampling trajectories (1/9)

2

63

392

Yilun Xu

@xuyilun2

2 years

Get ready to upgrade your diffusion models😻! Our #iclr2023 paper reduces variance in denoising score-matching, improving image quality, stability, and training speed. Experience the best image generation with current SOTA FID of 1.90 on CIFAR-10

6

67

381

Yilun Xu

@xuyilun2

3 months

Officially passed my PhD thesis defense today! I'm deeply grateful to my collaborators and friends for their support throughout this journey. Huge thanks to my amazing thesis committee: Tommi Jaakkola (advisor), @karsten_kreis , and @phillip_isola ! 🎓✨

18

13

239

Yilun Xu

@xuyilun2

2 years

Excited to share our #NeurIPS2022 paper "Poisson Flow Generative Models". The PFGM ODE achieves - SOTA performance in (continuous) normalizing flow family - Faster sampling speed than SDEs in diffusion models Paper: Code: … 1/n

4

33

219

Yilun Xu

@xuyilun2

1 year

In diffusion models, samplers are primarily ODE-centric, overlooking slower stochastic methods. However, we show that stochastic sampler can outperform previous samplers on Stable Diffusion, if we use stochasticity correctly! check out Restart Sampling:

2

34

174

Yilun Xu

@xuyilun2

1 year

I'll be in #ICML to present #PFGMpp (Thurs 1:30-3pm, Exhibit Hall 1 #545 ), and discuss with an awesome panel about new frontier of generative model at #SPIGM workshop (). Happy to chat about diffusion model, PFGM, or new physics-inspired generative models!

Yilun Xu

@xuyilun2

1 year

Excited to share PFGM++ #ICML2023 : a physics-inspired generative model unifying diffusion models & PFGM! By embedding N-dim data in N+D-dim space, we achieve: ✨ Flexible D for robustness & rigidity ✨ Median Ds outperform SOTA diffusion models (D-> inf)

3

15

77

1

18

92

Yilun Xu

@xuyilun2

1 year

Excited to share PFGM++ #ICML2023 : a physics-inspired generative model unifying diffusion models & PFGM! By embedding N-dim data in N+D-dim space, we achieve: ✨ Flexible D for robustness & rigidity ✨ Median Ds outperform SOTA diffusion models (D-> inf)

3

15

77

Yilun Xu

@xuyilun2

11 months

Restart is accepted by #NeurIPS23 , and gets deployed in the popular webui . Let's combine the best of SDE (better quality) and ODE (faster sampling) samplers!

GitHub - AUTOMATIC1111/stable-diffusion-webui: Stable Diffusion web UI

Stable Diffusion web UI. Contribute to AUTOMATIC1111/stable-diffusion-webui development by creating an account on GitHub.

github.com

Yilun Xu

@xuyilun2

1 year

In diffusion models, samplers are primarily ODE-centric, overlooking slower stochastic methods. However, we show that stochastic sampler can outperform previous samplers on Stable Diffusion, if we use stochasticity correctly! check out Restart Sampling:

2

34

174

1

11

71

Yilun Xu

@xuyilun2

8 months

Presenting "Restart Sampling for Improving Generative Processes" @ #NeurlPS2023 today! Poster #808 5pm-7pm Come by to chat about the fast sampling for diffusion models!

Yilun Xu

@xuyilun2

11 months

Restart is accepted by #NeurIPS23 , and gets deployed in the popular webui . Let's combine the best of SDE (better quality) and ODE (faster sampling) samplers!

1

11

71

2

11

64

Yilun Xu

@xuyilun2

9 months

non-IID sampling can promote diversity / mitigate memorization in diffusion models!

Gabriele Corso

@GabriCorso

9 months

New paper!🤗 Do all your samples from Stable Diffusion or Dall-E look very similar to each other? It turns out IID sampling is to blame! We study this problem and propose Particle Guidance, a technique to obtain diverse samples that can be readily applied to your diffusion model!

4

86

439

0

6

53

Yilun Xu

@xuyilun2

2 years

Check out our ICLR spotlight paper on constructing the **orthogonal classifiers** that enable/outperform baselines in three tasks : - Controlled style transfer - Domain adaptation with label shifts - Fairness Paper: Code:

1

2

40

Yilun Xu

@xuyilun2

1 year

Excited to share our latest paper which establishes a duality between generative models and physical processes 😃.

Ziming Liu

@ZimingLiu11

1 year

Generative models have been inspired by physics, but Eureka-type “inspirations” are mysterious. Is there a systematic way to convert physical processes to generative models? The answer is yes! This will largely augment design space of generative models.

2

54

231

1

2

33

Yilun Xu

@xuyilun2

3 months

@kohjingyu This may not necessarily be a binary problem (diffusion versus auto-regressive). It is indeed possible to integrate the strengths of the two into a single model through a novel training pipeline. Stay tuned for our new model designed to achieve this 🙂

4

1

32

Yilun Xu

@xuyilun2

11 months

Quanta magazine just released an article featuring our recent series of works on generative AI 😬

Quanta Magazine

@QuantaMagazine

11 months

Researchers are exploring whether “physics-inspired generative models” might offer more transparent and effective forms of artificial intelligence. Steve Nadis reports:

1

41

166

0

17

Yilun Xu

@xuyilun2

2 months

@karsten_kreis @phillip_isola Please find my PhD thesis here:

phd_thesis_yilun_xu.pdf

Shared with Dropbox

www.dropbox.com

1

0

14

Yilun Xu

@xuyilun2

29 days

Diffusion models transform the simple Gaussian into the complex and multimodal data distribution through an ODE. The ODE mapping necessarily needs to be highly complex, with strong curvature (see the middle figure). (2/9)

1

3

16

Yilun Xu

@xuyilun2

2 years

@JosephJacks_ @nearcyan Thanks! Our recent experiments shows that we can further achieve 100x - 200x speedup with no loss on image quality, with some improvements on sampling methods. Stay tuned 😉

0

1

16

Yilun Xu

@xuyilun2

29 days

For more information, please visit our project page: . Shout out to my awesome collaborators and advisors @GabriCorso , Tommi Jaakkola, @ArashVahdat and @karsten_kreis (9/9)

DisCo-Diff: Enhancing Continuous Diffusion Models with Discrete Latents

DisCo-Diff combines continuous diffusion models with learnable discrete latents, simplifying the ODE process and enhancing performance across various tasks.

research.nvidia.com

0

4

14

Yilun Xu

@xuyilun2

2 years

Happy to see the great application of V-information in the NLP domain😋

Swabha Swayamdipta

@swabhz

2 years

🎉🎉Super thrilled that our paper on Understanding Dataset Difficulty with V-usable information received an outstanding paper award at #ICML2022 !! 🥳Looking forward to the broader applications of this framework. It was a total delight working with my @allen_ai intern, @ethayarajh

16

29

349

1

0

15

Yilun Xu

@xuyilun2

2 years

code: joint work w/ @ShangyuanTong and Tommi Jaakkola

GitHub - Newbeeer/stf: Code for ICLR 2023 Paper, "Stable Target Field for Reduced Variance Score...

Code for ICLR 2023 Paper, "Stable Target Field for Reduced Variance Score Estimation in Diffusion Models” - Newbeeer/stf

github.com

1

3

14

Yilun Xu

@xuyilun2

29 days

An additional autoregressive model post-hoc the distribution of the discrete latent. The discrete latent captures global statistics in Euclidean space, such as layouts, shapes, and color variations. These statistics are complementary to semantics, such as class labels. (5/9)

1

0

11

Yilun Xu

@xuyilun2

2 years

Very detailed and educational blog of PFGM!

Ryan O'Connor

@r_o_connor

2 years

Stable Diffusion runs on physics-inspired Deep Learning. Researchers from MIT (first authors @ZimingLiu11 and @xuyilun2 ) have recently unveiled a new physics-inspired model that runs even faster! This introduction has everything you need to know 👇

2

34

162

0

1

12

Yilun Xu

@xuyilun2

11 months

@LucaAmb @NeurIPSConf The phase transition corresponds to the far/middle/near field studied in

Stable Target Field for Reduced Variance Score Estimation in...

Diffusion models generate samples by reversing a fixed forward diffusion process. Despite already providing impressive empirical results, these diffusion models algorithms can be further improved...

arxiv.org

0

3

12

Yilun Xu

@xuyilun2

29 days

Empirically, DisCo-Diff consistently improves model performance on several image synthesis tasks and molecular docking. It achieves the new state-of-the-art on ImageNet-64/ImageNet-128 with an ODE sampler. (6/9)

1

9

Yilun Xu

@xuyilun2

29 days

To this end, we augment diffusion model with learnable discrete latent, inferred with an encoder, and train diffusion model and encoder end-to-end. The encoder is encouraged to encode global discrete structure into the latent and help the denoiser to reconstruct the data (4/9).

1

9

Yilun Xu

@xuyilun2

29 days

We also test DisCo-Diff on molecular docking, building upon the DiffDock framework. We see that also in this domain discrete latents provide improvements, with the success rate on the full dataset increasing from 32.9% to 35.4% and from 13.9% to 18.5%. (7/9)

1

0

9

Yilun Xu

@xuyilun2

1 year

In practice, Restart beats SDE and ODE samplers in speed and quality on CIFAR-10 and ImageNet-64. Additionally, Restart achieves a better balance of text-image alignment, visual quality, and diversity on large-scale text-to-image Stable Diffusion! Code:

GitHub - Newbeeer/diffusion_restart_sampling: Code for NeurIPS 2023 paper "Restart Sampling for...

Code for NeurIPS 2023 paper "Restart Sampling for Improving Generative Processes" - Newbeeer/diffusion_restart_sampling

github.com

1

0

9

Yilun Xu

@xuyilun2

29 days

We believe DisCo-Diff could be further extended text-to-image/video generation, where we would expect discrete latent variables to offer complementary benefits to the text conditioning, similar to how discrete latents boost performance in our class-conditional experiments. (8/9)

6

0

7

Yilun Xu

@xuyilun2

2 years

Inspired by the electric field in physics, we interpret the data points as electrical charges on the z = 0 hyperplane in a space augmented with an additional dimension z The electric field lines transform data distribution into a uniform distribution on the large hemisphere. 3/n

1

3

7

Yilun Xu

@xuyilun2

2 months

@jon_barron That’s how we construct the prior distribution in PFGM (projecting uniform distribution on sphere to a hyperplane). To generate Gaussian, one could simply kick balls towards a cylinder in a uniform angle in an infinite-dimensional space as shown in PFGM++.

0

6

Yilun Xu

@xuyilun2

2 years

@alexjc Thanks for your interests! Our PFGM is different from Diffusion models: Diffusion models arise from thermodynamics, but PFGM is inspired by electrostatics. PFGM outperforms diffusion models in sample quality and sampling speed :), using similar architecture.

0

1

6

Yilun Xu

@xuyilun2

29 days

Conversely, using the known global discrete structure (e.g., index of modes) of data as input for diffusion models reduces the curvature of the ODE path (see right figure above). A key challenge remains: how to infer this discrete structure directly from the data? (3/9)

1

0

6

Yilun Xu

@xuyilun2

1 year

ODE samplers are fast but plateau in performance while SDE samplers deliver better samples at the cost of increased sampling time. We attribute this difference to sampling errors: ODE involve smaller discretization errors while stochasticity in SDE contracts accumulated errors.

1

0

6

Yilun Xu

@xuyilun2

2 years

We also design a backward ODE for sampling. The backward ODE transforms samples from uniform hemisphere to the data distribution. 5/n

1

5

Yilun Xu

@xuyilun2

1 year

Code:

GitHub - Newbeeer/pfgmpp: Code for ICML 2023 paper, "PFGM++: Unlocking the Potential of Physics-I...

Code for ICML 2023 paper, "PFGM++: Unlocking the Potential of Physics-Inspired Generative Models" - Newbeeer/pfgmpp

github.com

0

4

Yilun Xu

@xuyilun2

2 years

We learn the high dimensional version of the electric field, termed Poisson field, by neural networks. Specifically, we use a large batch to calculate the normalized field. 4/n

1

5

Yilun Xu

@xuyilun2

2 years

Our method achieves SOTA results on CIFAR10 dataset in flow family, faster sampling speed than SDEs in score-based and diffusion models, and more robustness. It can also scale to higher resolution dataset, e.g. LSUN bedroom 256x256. n/n

0

1

4

Yilun Xu

@xuyilun2

1 year

Joint work with @ZimingLiu11 @YonglongT @ShangyuanTong @tegmark and Tommi Jaakkola

1

0

4

Yilun Xu

@xuyilun2

1 year

Joint work w/ @Goodeat258 , Xiang Cheng, @YonglongT , @ZimingLiu11 and Tommi Jaakkola

0

4

Yilun Xu

@xuyilun2

1 year

Based on these findings, we propose a novel sampling algorithm called Restart in order to better balance discretization errors and contraction. The sampling method alternates between adding substantial noise in additional forward steps and strictly following a backward ODE.

1

0

4

Yilun Xu

@xuyilun2

10 months

@s_mandt I think other forward processes like PFGM++ and EDM already get FID smaller than the number in this paper on CIFAR-10?

1

0

1

Yilun Xu

@xuyilun2

11 months

In our updated version, we will show that the Restart sampling can also produce better samples in low-NFE regime (~20 NFE) on benchmarks and SD.

1

0

3

Yilun Xu

@xuyilun2

2 years

Just set up my twitter account. Sorry for the huge delay 😃 @baaadas @shengjia_zhao @StefanoErmon

David Duvenaud

@DavidDuvenaud

4 years

I really like this new paper on "Usable Information under Computational Constraints". It generalizes Shannon information to consider the ease of making predictions using a particular representation. by @baaadas , @shengjia_zhao , @StefanoErmon et al.

4

31

250

0

2

Yilun Xu

@xuyilun2

3 months

@nanliuuu @karsten_kreis @phillip_isola Thanks, Nan!

0

2

Yilun Xu

@xuyilun2

2 years

joint work with @ZimingLiu11 , @tegmark and Tommi Jaakkola 2/n

1

0

2

Yilun Xu

@xuyilun2

3 months

@GuangHeLee1 Thanks Guang-He 哥

0

2

Yilun Xu

@xuyilun2

2 years

@_rk_singhal Feel free to add it into the list :)

0

2

Yilun Xu

@xuyilun2

3 months

@hjian42 @karsten_kreis @phillip_isola Thanks, Hang!

0

2

Yilun Xu

@xuyilun2

2 years

@YunzhuLiYZ @UofIllinois @uofigrainger @IllinoisCS @StanfordSVL @jiajunwu_cs @drfeifei Congrats Yunzhu!!

1

0

1

Yilun Xu

@xuyilun2

3 months

@DibyajyotiAch04 Thank you! I will share the link soon!

0

1

Yilun Xu

@xuyilun2

1 year

@janekm Thanks for the information! We plan to PR to diffuser repo, and will also take a look at automatic1111!

0

1

Yilun Xu

@xuyilun2

3 months

@DrYangSong Thank you Yang!

0

1

Yilun Xu

@xuyilun2

3 months

@chenlin_meng Thank you Chenlin!

0

1

Yilun Xu

@xuyilun2

3 months

@SahajGarg6 Thanks Sahaj!

0

1

Yilun Xu

@xuyilun2

2 months

@DibyajyotiAch04 Please find it here:

phd_thesis_yilun_xu.pdf

Shared with Dropbox

www.dropbox.com

0

1

Yilun Xu

@xuyilun2

11 months

@timudk Thanks Tim, we will look into it :)

0

1

Yilun Xu

@xuyilun2

2 years

@menghua_wu 😍cute dogs

0

1

Yilun Xu

@xuyilun2

2 years

awesome collaborators @hehaodele , Tianxiao Shen and Tommi Jaakkola

0

1

Yilun Xu

@xuyilun2

1 year

@YonglongT Congrats Cambridge A Long!

1

0

1

Yilun Xu

@xuyilun2

1 year

@Pavel_Izmailov @andrewgwils @ylecun @kchonyc @FeiziSoheil @sirbayes @sainingxie Congrats Pavel!

1

0

1

Yilun Xu

@xuyilun2

3 months

@elmelis @karsten_kreis @phillip_isola Thanks, David!

0

1

Yilun Xu

@xuyilun2

1 year

@madebyollin Thanks for pointing this out, unfortunately that's a limitation on current method and one has to construct separate reference batches for different conditions. But please try STF if you can maintain these batches! (naive DSM is STF with batch size=1)

0

1

Yilun Xu

@xuyilun2

3 months

@adityagrover_ Thank you, Aditya!

0

1