Paul Liang @pliang279 Twitter profile | Pikagi

Pikagi

Paul Liang

@pliang279

4,613

Followers

911

Following

223

Media

3,543

Statuses

PhD student @mldcmu @SCSatCMU . Foundations of multimodal learning & applications in social AI, NLP, and healthcare with @lpmorency and @rsalakhu .

Pittsburgh, PA

https://t.co/4MOotEoFYU

Joined July 2012

Don't wanna be here? Send us removal request.

Pinned Tweet

@pliang279

Paul Liang

1 year

I'm on the faculty market in fall 2023! I work on foundations of multimodal ML applied to NLP, socially intelligent AI & health. My research & teaching If I could be a good fit for your department, I'd love to chat at #ACL2023NLP & #ICML2023 DM/email me!

2

62

296

Last Seen Profiles

@MiNimmm2006

@edwardbarsamian

@doremifaso64

@CoachJeffGarner

@michelledenixse

@Abdlrahmanhadad

@zak_file

@BirdOnMyTimelin

@Theeegodson

@D_angeRou

@fandomclexa

@Bozankazanova

@TheBoyThe6

@kaahfps

@stwmaniax

@stwmaniax

@femmeduartar

@srr3i

@Asmaa_Elarabi88

@angelsbsnss_

@AliGom3aaa

@alaouf

@PhDtoDO

@alaouf

@bokeplokalmalam

@wandaklee

@21_1k1l

@fotor_com

@MC481ya

@nyshaVLR

@aparnapkin

@TigerTalk_TV

@mayouKOURO

@mufti_sajid_

@DianaSu23218299

@wellredwinemag

@pliang279

Paul Liang

8 months

Despite the successes of contrastive learning (eg CLIP), they have a fundamental limitation - it can only capture *shared* info between modalities, and ignores *unique* info To fix it, a thread for our #NeurIPS2023 paper w Zihao Martin @james_y_zou @lpmorency @rsalakhu :

Tweet media one

2

81

422

@pliang279

Paul Liang

2 years

If recent models like DALL.E, Imagen, CLIP, and Flamingo have you excited, check out our upcoming #CVPR2022 tutorial on Multimodal Machine Learning - next monday 6/20 9am-1230pm slides, videos & a new survey paper will be posted soon after the tutorial!

Tweet media one

6

70

337

@pliang279

Paul Liang

1 year

As PhD visit days are coming up, I'd like to share this collated resource for prospective & current PhDs, covering how to choose advisors & schools, advice for research, teaching, fellowships, networking & more Credit to the original authors of each link!

Tweet media one

4

45

325

@pliang279

Paul Liang

2 years

Check out slides & video recordings of our recent tutorials on multimodal machine learning at CVPR 2022 and NAACL 2022: video: slides and readings:

Tweet media one

0

73

284

@pliang279

Paul Liang

2 years

Really excited to release the video of my guest lecture on Multimodal Deep Learning for CMU's Deep Learning class @rsalakhu @mldcmu It covers 5 fundamental concepts in multimodal ML: representation, alignment, reasoning, translation & co-learning Youtube:

Tweet card media

Multimodal Deep Learning - CMU 10707 Guest Lecture

Lecture 21: Multimodal Deep Learning (10707 Advanced Deep Learning, Carnegie Mellon University)Topics: Research and Technical Challenges in Multimodal Deep L...

www.youtube.com

4

59

274

@pliang279

Paul Liang

9 months

Multimodal AI studies the info in each modality & how it relates or combines with other modalities. This past year, we've been working towards a **foundation** for multimodal AI: I'm excited to share our progress at #NeurIPS2023 & #ICMI2023 : see long 🧵:

Tweet media one

2

78

272

@pliang279

Paul Liang

2 years

[11877 Advanced Topics in Multimodal ML] In week 5’s session, the class aimed to define a taxonomy of multimodal reasoning: the (hierarchical) composition of unimodal and multimodal evidences into higher-level abstract concepts for prediction. Notes here:

Tweet media one

Tweet media two

Tweet media three

Tweet media four

2

49

245

@pliang279

Paul Liang

1 year

This semester, @lpmorency and I are teaching 2 new graduate seminars @LTIatCMU @mldcmu . The first, 11-877 Advanced Topics in Multimodal Machine Learning, focuses on open research questions and recent theoretical & empirical advances in multimodal ML:

Tweet media one

3

40

237

@pliang279

Paul Liang

4 years

Follow our course 11-777 Multimodal Machine Learning, Fall 2020 @ CMU @LTIatCMU with new content on multimodal RL, bias and fairness, and generative models. All lectures will be recorded and uploaded to Youtube.

4

57

200

@pliang279

Paul Liang

5 years

I am compiling a reading list for multimodal machine learning () containing important papers, workshops, tutorials, and courses, updated for #ICML2019 and #CVPR2019 ! @mldcmu @LTIatCMU

6

36

189

@pliang279

Paul Liang

1 year

Multimodal models like VilBERT, CLIP & transformers are taking over by storm! But do we understand what they learn? At #ICLR2023 we're presenting MultiViz, an analysis framework for model understanding, error analysis & debugging.

Tweet media one

4

29

192

@pliang279

Paul Liang

2 years

Are you working on multimodal tasks and can't decide on a model? Check out HighMMT -our attempt at a single Transformer model with shared parameters for sentiment, emotion, humor, disease, robot pose prediction & more! paper: code:

Tweet media one

4

40

181

@pliang279

Paul Liang

2 years

I'm compiling awesome advice I've found most useful while navigating my CS PhD . Contains sections for prospective and current students - credit goes out to the original authors of each link! It's what I wish I had seen when applying and starting my PhD 🎉

Tweet media one

0

33

166

@pliang279

Paul Liang

3 years

Excited to present MultiBench, a large-scale benchmark for multimodal representation learning across affective computing, healthcare, robotics, finance, HCI, and multimedia domains at #NeurIPS2021 benchmarks track! 🎉 paper: code:

Tweet media one

3

35

156

@pliang279

Paul Liang

1 year

Extremely honored to have received the teaching award - check out our publicly available CMU courses and resources on multimodal ML (MML) and artificial social intelligence (ASI): MML: Advanced MML: ASI:

@mldcmu

Machine Learning Dept. at Carnegie Mellon

1 year

Paul Liang ( @pliang279 ) received the 2023 Graduate Student Teaching Award "for incredible work in designing and teaching several new courses in Multimodal Machine Learning and Artificial Social Intelligence, general excellence in teaching, and excellence in student mentorship."

Tweet media one

5

4

33

11

11

150

@pliang279

Paul Liang

2 years

Are you working on federated learning over heterogeneous data? Use Vision Transformers as a backbone! In our upcoming #CVPR2022 paper, we perform extensive experiments demonstrating the effectiveness of ViTs for FL: paper: code:

Tweet media one

Tweet media two

2

32

147

@pliang279

Paul Liang

14 days

Excited to release HEMM (Holistic Evaluation of Multimodal Foundation Models), the largest and most comprehensive evaluation for multimodal models like Gemini, GPT-4V, BLIP-2, OpenFlamingo, and more. HEMM contains 30 datasets carefully selected and categorized based on: 1. The

Tweet media one

2

35

139

@pliang279

Paul Liang

3 years

Extremely honored to have received a Facebook PhD Fellowship to support my research in socially intelligent AI! I am indebted to my advisors @lpmorency @rsalakhu at @mldcmu and my collaborators @brandondamos @_rockt @egrefen at @facebookai .

Tweet card media

Announcing the recipients of the 2021 Facebook Fellowship awards - Meta Research | Meta Research

The Facebook Fellowship program provides awards to PhD candidates conducting research on important topics across computer science and engineering, such...

research.facebook.com

10

8

135

@pliang279

Paul Liang

1 year

Excited that HighMMT - our attempt at a single multimodal transformer model with shared parameters for many modalities including images, videos, sensors, sets, tables & more, was accepted at TMLR: paper: code:

Tweet media one

@TmlrPub

Accepted papers at TMLR

1 year

High-Modality Multimodal Transformer: Quantifying Modality & Interaction Heterogeneity for High-M... Paul Pu Liang, Yiwei Lyu, Xiang Fan et al.. Action editor: Brian Kingsbury. #multimodal #modality #gestures

0

6

13

3

29

114

@pliang279

Paul Liang

2 years

[11877 Advanced Topics in Multimodal ML] In week 6’s session, the class discussed various challenges and approaches in modeling memory and long-term interactions in multimodal tasks. Notes here:

Tweet media one

Tweet media two

Tweet media three

Tweet media four

1

15

116

@pliang279

Paul Liang

4 years

Check out the recorded lecture videos for CMU 11-777 Multimodal Machine Learning @LTIatCMU ! with new content on multimodal RL, generative models, alignment, and upcoming guest lectures on #RoboNLP , fairness & multilingual #NLP !

Tweet card media

Videos by Louis-Philippe Morency, associate professor at CMU. This channel includes lectures from CMU's Multimodal Machine Learning course (11-777).

www.youtube.com

0

37

114

@pliang279

Paul Liang

2 years

[11877 Advanced Topics in Multimodal ML] In week 9’s session, the class discussed insights about how the brain performs multimodal perception & integration, and brainstormed possible directions toward brain-inspired multimodal models. Notes here:

Tweet media one

Tweet media two

Tweet media three

Tweet media four

3

24

115

@pliang279

Paul Liang

3 years

dressed up as our PhD advisor @rsalakhu for (belated) halloween! cos there’s nothing scarier than your advisor 👻🥳😝 jk Russ is the best 🎉🍾❤️

Tweet media one

1

5

115

@pliang279

Paul Liang

2 years

Hugely excited to release our mammoth survey paper on multimodal ML to support our courses and tutorials at CMU and international conferences: paper: tutorial slides and videos: CMU multimodal ML course:

2

28

111

@pliang279

Paul Liang

2 years

[11877 Advanced Topics in Multimodal ML] In week 13, the class discussed challenges and techniques for interpreting and explaining multimodal models and data, as well as their evaluation. Notes here:

Tweet media one

Tweet media two

Tweet media three

Tweet media four

1

22

97

@pliang279

Paul Liang

1 year

icymi @ #ICML2023 , the latest multimodal ML tutorial slides are posted here: along with a reading list of important work covered in the tutorial, as well as slides and videos for previous versions (more application focused)

@pliang279

Paul Liang

1 year

If you're attending #ICML2023 don't miss our tutorial on multimodal ML (w @lpmorency ) Content: 1. Three key principles of modality heterogeneity, connections & interactions 2. Six technical challenges 3. Open research questions Monday July 24, 930 am Hawaii time, Exhibit Hall 2

Tweet media one

Tweet media two

1

10

82

3

25

99

@pliang279

Paul Liang

4 years

If you're applying for PhD @SCSatCMU or MLT @LTIatCMU , you can get your application reviewed by current grad students! To participate, submit your application materials by Nov 9 - we particularly encourage underrepresented groups to apply 🎉 @mldcmu

2

23

98

@pliang279

Paul Liang

4 years

If @OpenAI 's CLIP & DALL·E has gotten you interested in multimodal learning, check out a reading list (w code) here covering various modalities (language, vision, speech, video, touch) & applications (QA, dialog, reasoning, grounding, healthcare, robotics)

Tweet card media

GitHub - pliang279/awesome-multimodal-ml: Reading list for research topics in multimodal machine...

Reading list for research topics in multimodal machine learning - pliang279/awesome-multimodal-ml

1

22

91

@pliang279

Paul Liang

1 year

If GPT4's got you excited about multimodal, but you want to know the technical details of *what is multimodal*, *why is it hard* & *what is next*, with public resources, code & models, check out lecture slides & videos of our multimodal ML course @ CMU:

Tweet media one

Tweet media two

1

20

89

@pliang279

Paul Liang

3 years

Excited to share our new work on measuring and mitigating social biases in pretrained language models, to appear at #ICML2021 ! with Chiyu Wu, @lpmorency , @rsalakhu @mldcmu @LTIatCMU check it out here: paper: code:

Tweet media one

1

25

87

@pliang279

Paul Liang

3 years

check out our multimodal ML course @LTIatCMU - all lecture videos and course content available online!

1

10

86

@pliang279

Paul Liang

2 years

If you're coming to #NAACL2022 drop by our workshop on Multimodal AI, now in its 4th edition! We have invited speakers covering multimodal learning for embodied AI, virtual reality, robotics, HCI, healthcare, & education! July 15, 9am-4pm

Tweet media one

Tweet media two

Tweet media three

3

20

87

@pliang279

Paul Liang

8 months

Excited to attend #NeurIPS2023 this week! Find me to chat about the foundations of multimodal machine learning, multisensory foundation models, interactive multimodal agents, and their applications. I'm also on the academic job market, you can find my statements on my website:

Tweet media one

1

11

87

@pliang279

Paul Liang

4 years

found this gem of a reading list for NLP: focuses on biases, fairness, robustness, and understanding of NLP models. collected by @kaiwei_chang @UCLA #NLProc

Tweet media one

2

18

83

@pliang279

Paul Liang

5 years

some exciting recent work in self-supervised multimodal learning including VideoBERT (), ViLBERT (), and VisualBERT (). for more papers in multimodal representation learning, check out

Tweet media one

Tweet media two

Tweet media three

0

21

81

@pliang279

Paul Liang

1 year

If you're attending #ICML2023 don't miss our tutorial on multimodal ML (w @lpmorency ) Content: 1. Three key principles of modality heterogeneity, connections & interactions 2. Six technical challenges 3. Open research questions Monday July 24, 930 am Hawaii time, Exhibit Hall 2

Tweet media one

Tweet media two

1

10

82

@pliang279

Paul Liang

5 years

Excited to present Deep Gamblers: Learning to Abstain with Portfolio Theory at #NeurIPS2019 ! Strong results for uncertainty estimation, learning from noisy data and labels. with Ziyin, Zhikang, @rsalakhu , LP, Masahito paper: code:

Tweet media one

Tweet media two

Tweet media three

Tweet media four

1

14

79

@pliang279

Paul Liang

2 years

[11877 Advanced Topics in Multimodal ML] In week 11, the class formalized a taxonomy of dataset and model biases (social bias, annotator bias, shortcuts, spurious correlations) and proposed solutions to mitigate them in multimodal settings. Notes here:

Tweet media one

Tweet media two

Tweet media three

Tweet media four

1

23

78

@pliang279

Paul Liang

1 year

A few weeks ago @lpmorency and I wrapped up this semester's offerings of 2 new graduate seminars @LTIatCMU @mldcmu . We're releasing all course content, discussion questions, and readings here for the public to enjoy:

Tweet media one

Tweet media two

2

13

65

@pliang279

Paul Liang

2 years

I gave a talk about some of our recent work on multimodal representation learning and their applications in healthcare last week at @MedaiStanford check out the video recording here: links to papers and code:

Tweet card media

MedAI #56: Fundamentals of Multimodal Representation Learning | Paul...

Title: Fundamentals of Multimodal Representation Learning - Towards Generalization and QuantificationSpeaker: Paul Pu LiangAbstract:In recent years, the ques...

www.youtube.com

@MedaiStanford

MedAI Group

2 years

This week, @pliang279 from CMU will be joining us to talk about fundamentals of multimodal representation learning. Catch it at 1-2pm PT this Thursday on Zoom! Subscribe to #ML #AI #medicine #healthcare

Tweet media one

0

7

32

0

7

61

@pliang279

Paul Liang

6 years

Come check out our talks and posters at #NeurIPS tomorrow! 1. Learning Multimodal Representations with Factorized Deep Generative Models @ Bayesian Deep Learning workshop with Hubert Tsai, Amir Zadeh, LP Morency, and @rsalakhu @mldcmu @LTIatCMU @SCSatCMU

Tweet media one

2

15

58

@pliang279

Paul Liang

5 years

friends interested in multimodal learning: I've updated my reading list with the latest papers (+code) and workshops at #NeurIPS2019 . cool new papers spanning multimodal RL, few-shot video generation, multimodal pretraining, and emergent communication!

Tweet card media

GitHub - pliang279/awesome-multimodal-ml: Reading list for research topics in multimodal machine...

Reading list for research topics in multimodal machine learning - pliang279/awesome-multimodal-ml

1

13

58

@pliang279

Paul Liang

4 years

Excited that our paper on efficient sparse embeddings for large vocabulary sizes was accepted at #ICLR2021 ! strong results on text classification, language modeling, recommender systems with up to 44M items and 15M users! w Manzil, Yuan, Amr @GoogleAI

Tweet card media

Anchor & Transform: Learning Sparse Embeddings for Large Vocabularies

Learning continuous representations of discrete objects such as text, users, movies, and URLs lies at the heart of many applications including language and user modeling. When using discrete...

@pliang279

Paul Liang

4 years

Anchor & Transform: efficiently learn embeddings using a set of dense anchors and sparse transformations! + statistical interpretation as a Bayesian nonparametric prior which further learns an optimal number of anchors w awesome collaborators @GoogleAI

Tweet media one

Tweet media two

1

2

26

3

9

55

@pliang279

Paul Liang

2 years

Follow 11-777 Multimodal Machine Learning @ CMU, now in its 12th edition! A completely revamped version based on our tutorials on multimodal ML and new taxonomy of technical challenges: all slides & videos are publicly available

Tweet media one

Tweet media two

Tweet media three

0

16

56

@pliang279

Paul Liang

5 years

Excited to announce the 2nd workshop on multimodal language @ #ACL2020 ! We welcome submissions in all areas of human language, multimodal ML, multimedia, affective computing, and applications! w/ fantastic speakers: @radamihalcea @rsalakhu @ehsan_hoque

Tweet media one

Tweet media two

Tweet media three

2

15

53

@pliang279

Paul Liang

4 years

With many grad student visit days happening this month, @andrewkuznet has written an educational post on the ML @CMU blog on questions to ask prospective Ph.D. advisors! Please share with your friends who are attending visit days all around the world!

Tweet card media

Questions to Ask a Prospective Ph.D. Advisor on Visit Day, With Thorough and Forthright Explanations

These are my (@andrewkuznet) opinions, formed by being a SCS PhD student at Carnegie Mellon University in the US. This post is meant as a followup to a poster I made in 2019 with the help of many...

blog.ml.cmu.edu

0

12

52

@pliang279

Paul Liang

3 years

We ( @lpmorency Amir and I) are organizing a new seminar course on advanced topics in multimodal ML: It will primarily be reading and discussion-based. We've come up with a list of open research questions and will post discussion highlights every friday!

Tweet media one

Tweet media two

2

5

53

@pliang279

Paul Liang

4 years

Really proud to be a student @SCSatCMU @mldcmu ! Taking classes and doing research with Turing award winners and leaders in their fields, achieving gender parity in CS, and in the midst of amazing people working on important problems in fairness, interpretability, & ethics!

0

2

50

@pliang279

Paul Liang

2 years

If you weren't able to join us for #CVPR2022 , we'll be giving an updated tutorial on multimodal machine learning at #NAACL2022 in Seattle this Sunday, July 10, 2:00–5:30pm. slides and videos are already posted here:

Tweet media one

Tweet media two

@pliang279

Paul Liang

2 years

If recent models like DALL.E, Imagen, CLIP, and Flamingo have you excited, check out our upcoming #CVPR2022 tutorial on Multimodal Machine Learning - next monday 6/20 9am-1230pm slides, videos & a new survey paper will be posted soon after the tutorial!

Tweet media one

6

70

337

0

9

48

@pliang279

Paul Liang

3 years

My advisor @lpmorency is finally on twitter! Follow him to stay up to date with awesome work in multimodal ML, NLP, human-centric ML, human behavior analysis, and applications in healthcare and education coming out of the MultiComp Lab @LTIatCMU @mldcmu

Tweet media one

4

4

48

@pliang279

Paul Liang

2 years

[11877 Advanced Topics in Multimodal ML] In week 14, the class discussed technical challenges in multimodal generation, the evaluation of generation quality, and potential ethical issues of generative models. Notes here:

Tweet media one

Tweet media two

Tweet media three

Tweet media four

1

5

47

@pliang279

Paul Liang

2 years

[11877 Advanced Topics in Multimodal ML] In week 15, the class discussed challenges in generalization to a large number of modalities and tasks, with a particular focus on low-resource modalities and robustness to noisy and missing modalities. Notes here:

Tweet media one

Tweet media two

Tweet media three

Tweet media four

1

10

46

@pliang279

Paul Liang

3 years

Check out our #NAACL2021 paper on StylePTB: a compositional benchmark for fine-grained controllable text style transfer! with Yiwei, @hai_t_pham , Ed Hovy, Barnabas, @rsalakhu @lpmorency @mldcmu @LTIatCMU paper: code: a thread:

Tweet media one

3

16

44

@pliang279

Paul Liang

5 years

friends at #ICLR2019 , we are presenting our poster on "Learning Factorized Multimodal Representations" at 430pm today! paper: with Hubert, Amir, LP Morency, @rsalakhu @mldcmu @LTIatCMU

Tweet media one

0

5

39

@pliang279

Paul Liang

3 years

With many grad student visit days happening this month, it's time to pull up this blog post on @mlcmublog : *Questions to ask prospective Ph.D. advisors* Please share with your friends who are attending visit days all around the world! by @andrewkuznet

Tweet card media

Questions to Ask a Prospective Ph.D. Advisor on Visit Day, With Thorough and Forthright Explanations

These are my (@andrewkuznet) opinions, formed by being a SCS PhD student at Carnegie Mellon University in the US. This post is meant as a followup to a poster I made in 2019 with the help of many...

blog.ml.cmu.edu

1

8

35

@pliang279

Paul Liang

4 months

As prospective PhD student visit days are happening around the world, I would like to share a valuable resource @andrewkuznet has written on the @mlcmublog : **Questions to Ask a Prospective Ph.D. Advisor on Visit Day, With Thorough and Forthright Explanations**

Tweet media one

1

3

31

@pliang279

Paul Liang

9 months

If your downstream task data is quite different from your pretraining data, make sure you check out our new approach *Difference-Masking* at #EMNLP2023 findings. Excellent results on classifying citation networks, chemistry text, social videos, TV shows etc. see thread below:

@SNAT02792153

Syeda Nahida Akter

9 months

In continued pretraining, how can we choose what to mask when the pretraining domain differs from the target domain? In our #EMNLP2023 paper, we propose Difference-Masking to address this problem and boost downstream task performance! Paper:

Tweet media one

1

16

56

0

6

31

@pliang279

Paul Liang

5 years

running an ml research group in 2019: prof juggling post-docs juggling phd students juggling masters juggling undergrads

@pickover

Cliff Pickover

5 years

Juggler juggling jugglers. Wow.

27

593

2K

0

0

30

@pliang279

Paul Liang

1 year

Happening in ~2 hours at #ICML2023 930am @ exhibit hall 2 Also happy to chat about - understanding multimodal interactions and modeling them - models for many diverse modalities esp beyond image+text - Applications in health, robots, education, social intelligence & more DM me!

@pliang279

Paul Liang

1 year

If you're attending #ICML2023 don't miss our tutorial on multimodal ML (w @lpmorency ) Content: 1. Three key principles of modality heterogeneity, connections & interactions 2. Six technical challenges 3. Open research questions Monday July 24, 930 am Hawaii time, Exhibit Hall 2

Tweet media one

Tweet media two

1

10

82

1

1

30

@pliang279

Paul Liang

5 years

New ML @CMU blog post on understanding the behavior of gradient descent on nonconvex functions, by Chirag Gupta!

Tweet media one

Tweet media two

@eaplatanios

Anthony Platanios

5 years

New post on ML @CMU blog on path length bounds for gradient descent, written by Chirag Gupta!

0

2

10

0

5

30

@pliang279

Paul Liang

5 years

this is a wonderful post: about the important conversation around attention and its interpretation in NLP. main takeaway: be careful in interpreting attention weights as explanations, and attention should not be treated as justification for a decision.

Thoughts on “Attention is Not Not Explanation”

Sarah Wiegreffe and Yuval Pinter have a nice new paper out with an undeniably fun title: “attention is not not explanation”, continuing an…

0

5

28

@pliang279

Paul Liang

5 years

Congrats to @roboVisionCMU @CMU_Robotics for winning the #CVPR2019 best paper award! () For the second year in a row, they won the best paper/student paper with a paper **not** primarily about neural net architectures! #CVPR2018 :

Tweet card media

Total Capture: A 3D Deformation Model for Tracking Faces, Hands, and Bodies

We present a unified deformation model for the markerless capture of multiple scales of human movement, including facial expressions, body motion, and hand gestures. An initial model is generated...

0

1

27

@pliang279

Paul Liang

4 years

Follow ML @CMU blog @mlcmublog for your weekly dose of ML research, conference highlights, broad surveys of research areas, and tutorials! For starters, check out our recent post on best practices for real-world data analysis! @mldcmu @LTIatCMU @SCSatCMU

0

3

27

@pliang279

Paul Liang

4 years

excited to present our paper on studying biases in sentence encoders at #acl2020nlp : web: code: also happy to take questions during the live Q&A sessions: July 7 (14:00-15:00, 17:00-18:00 EDT) w Irene, Emily, YC, @rsalakhu , LP

Tweet media one

Tweet media two

Tweet media three

Tweet media four

0

5

28

@pliang279

Paul Liang

2 years

Do AI models know if an object can be easily broken💔? or melts at high heat🔥? Check out PACS: a new audiovisual question-answering dataset for physical commonsense reasoning and new models at #ECCV2022 this week: paper: video:

7

5

27

@pliang279

Paul Liang

5 years

friends at CMU, come check out the poster presentations for 10-708 Probabilistic Graphical Models, Tuesday 4/30 3-5pm at NSH atrium! projects cover theories and applications of pgms in nlp, rl, vision, graphs, healthcare, and more! @rl_agent @alshedivat @_xzheng @mldcmu

1

3

26

@pliang279

Paul Liang

4 years

Anchor & Transform: efficiently learn embeddings using a set of dense anchors and sparse transformations! + statistical interpretation as a Bayesian nonparametric prior which further learns an optimal number of anchors w awesome collaborators @GoogleAI

Tweet media one

Tweet media two

1

2

26

@pliang279

Paul Liang

2 years

check out tutorial slides and reading resources here: recorded tutorial videos will be uploaded soon #CVPR2022

Tweet media one

Tweet media two

@pliang279

Paul Liang

2 years

If recent models like DALL.E, Imagen, CLIP, and Flamingo have you excited, check out our upcoming #CVPR2022 tutorial on Multimodal Machine Learning - next monday 6/20 9am-1230pm slides, videos & a new survey paper will be posted soon after the tutorial!

Tweet media one

6

70

337

1

6

27

@pliang279

Paul Liang

2 years

Heading to #NeurIPS2022 - message me if you wanna watch the world cup or chat about multimodal machine learning, socially intelligent AI, and their applications in healthcare and education (in that order ⚽,🤖) My collaborators and I will be presenting the following papers:

1

3

27

@pliang279

Paul Liang

5 years

friends at #CVPR2019 , we're presenting Social-IQ: A Question Answering Benchmark for Artificial Social Intelligence on Thursday, June 20 @ Oral Session 3-1B, Grand Ballroom paper: data: w Amir, Michael, Edmund, LP @mldcmu @LTIatCMU

Tweet media one

Tweet media two

Tweet media three

Tweet media four

1

8

25

@pliang279

Paul Liang

2 years

@lpmorency @LTIatCMU @mldcmu @SCSatCMU This tutorial will cover 6 core challenges in multimodal ML: representation, alignment, reasoning, transference, generation, and quantification. Recent advances will be presented through the lens of this revamped taxonomy, along with future perspectives.

Tweet media one

0

3

25

@pliang279

Paul Liang

4 years

many cool papers on multimodal learning at #CVPR2020 , check them out (and more) here!

Tweet card media

GitHub - pliang279/awesome-multimodal-ml: Reading list for research topics in multimodal machine...

Reading list for research topics in multimodal machine learning - pliang279/awesome-multimodal-ml

1

3

24

@pliang279

Paul Liang

2 years

found a cute @Pittsburgh corner on r/place! ft @CarnegieMellon @steelers @PittTweet also ft some teams involved in a basketball game or something? not too sure.. 😝

Tweet media one

1

1

24

@pliang279

Paul Liang

5 years

friends at #AAAI19 come check out our spotlight talks and posters! 1. Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalities, 2pm, Coral 1 with @hai_t_pham , @Tom_Manzini , LP Morency, Barnabás Póczos @mldcmu @LTIatCMU @SCSatCMU

Tweet media one

Tweet media two

Tweet media three

1

4

24

@pliang279

Paul Liang

1 year

We'll be presenting the following at #ICLR2023 : Check out MultiViz, which answers: 1. what we should be interpreting in multimodal models, 2. how we can interpret them accurately, 3. how we can evaluate interpretability through real-world user studies:

@pliang279

Paul Liang

1 year

Multimodal models like VilBERT, CLIP & transformers are taking over by storm! But do we understand what they learn? At #ICLR2023 we're presenting MultiViz, an analysis framework for model understanding, error analysis & debugging.

Tweet media one

4

29

192

1

4

24

@pliang279

Paul Liang

2 years

@rsalakhu @mldcmu Many of the materials are based on the 2 full courses on Multimodal ML and Advanced Topics in Multimodal ML @ CMU. Check them out here! Multimodal ML: Lecture videos: Advanced Topics:

Tweet media one

Tweet media two

0

7

24

@pliang279

Paul Liang

2 years

starting soon at 9am in great hall B! #CVPR2022

Tweet media one

@pliang279

Paul Liang

2 years

If recent models like DALL.E, Imagen, CLIP, and Flamingo have you excited, check out our upcoming #CVPR2022 tutorial on Multimodal Machine Learning - next monday 6/20 9am-1230pm slides, videos & a new survey paper will be posted soon after the tutorial!

Tweet media one

6

70

337

0

5

22

@pliang279

Paul Liang

4 years

O(n) Transformer attention mechanism for long sequences: State of the art results with theory

Tweet media one

1

1

22

@pliang279

Paul Liang

4 years

A reminder to submit your work in multimodal ML, language, vision, speech, multimedia, affective computing, and applications to the workshop on multimodal language @ #ACL2020 ! with fantastic speakers: @radamihalcea @rsalakhu @ehsan_hoque @YejinChoinka

Tweet media one

Tweet media two

Tweet media three

1

3

23

@pliang279

Paul Liang

4 years

Join us this Friday for the first workshop on tensor networks at #NeurIPS2020 , with a fantastic lineup of speakers: @AmnonShashua @AnimaAnandkumar @oseledetsivan @yuqirose @jenseisert @fverstraete @giactorlai and 30 accepted papers!

Tweet media one

Tweet media two

@ccaiafa

Cesar Caiafa

4 years

Amazing speakers in our Workshop on Quantum Tensor Networks in Machine Learning at Neurisp2020. Please join us this Fri at #tensornetwors #quantum #machinelearning #neurisp2020 @XiaoYangLiu10 @JacobBiamonte @pliang279 @nadavcohen @sleichen

1

7

21

0

8

22

@pliang279

Paul Liang

10 months

If you're at #ICCV2023 , check out our new resource of lecture slides with speaker audio & videos. A step towards training and evaluating AI-based educational tutors that can answer and retrieve lecture content based on student questions! @ Friday 2:30-4:30pm Room Nord - 011

Tweet media one

2

3

21

@pliang279

Paul Liang

5 years

Found this gem submitted to #ICLR2020

1

1

21

@pliang279

Paul Liang

2 years

[11877 Advanced Topics in Multimodal ML] In week 10, the class discussed challenges in representation, scalability, and evaluation of multimodal learning from a large number of modalities, especially diverse ones beyond language & vision. Notes here:

Tweet media one

Tweet media two

Tweet media three

Tweet media four

1

6

21

@pliang279

Paul Liang

2 years

Nothing has excited me more than collaborating with and advising great students during my PhD. I've learned so much from them and I'm hugely excited to watch them embark on their new research agendas as incoming PhD students - follow all of them here for more exciting new ideas!

1

0

21

@pliang279

Paul Liang

6 years

2. Relational Attention Networks via Fully-Connected CRFs @ Bayesian Deep Learning workshop #NeurIPS2018 with Ziyin Liu, Junxiang Chen, Masahito Ueda @mldcmu @LTIatCMU @SCSatCMU

Tweet media one

1

5

20

@pliang279

Paul Liang

4 years

Our work on investigating competitive influences on emergent communication in multi-agent teams, to appear at #aamas2020 . paper: code: @rsalakhu @SatwikKottur @mldcmu @LTIatCMU

Tweet card media

GitHub - pliang279/Competitive-Emergent-Communication: [AAMAS 2020] On Emergent Communication in...

[AAMAS 2020] On Emergent Communication in Competitive Multi-Agent Teams - pliang279/Competitive-Emergent-Communication

@rsalakhu

Russ Salakhutdinov

4 years

On Emergent Communication in Competitive Multi-Agent Teams: External competitive influence leads to faster emergence of communicative languages that are more informative and compositional: #aamas2020 w/t @pliang279 , J. Chen, LP Morency, S. Kottur

Tweet media one

1

14

72

0

4

21

@pliang279

Paul Liang

2 years

@rsalakhu @lpmorency @mldcmu @LTIatCMU @gchhablani_ @hanzhao_ml @kunkzhang Vision-language models, despite their size, still struggle on compositional generalization benchmarks like Winoground. We show that incorporating structure in the attention alignment maps is a promising way to fine-tune these models for compositionality:

Tweet media one

0

3

20

@pliang279

Paul Liang

1 year

@lpmorency @LTIatCMU @mldcmu The second, 11-866 Artificial Social Intelligence, studies the interdisciplinary science and implications of socially intelligent AI that can perceive, reason, and interact in social situations with humans.

Tweet media one

1

3

19

@pliang279

Paul Liang

4 years

The ML @CMU blog has a new twitter account @mlcmublog ! Follow them for the latest machine learning research and educational blog posts coming out of @mldcmu @LTIatCMU @CMU_Robotics @SCSatCMU

@mlcmublog

ML@CMU

4 years

How many heads does multi-head attention need? Work from CMU shows that a large number of heads can be pruned at test time - in some cases even a single head is enough. New blog post by @pmichelX , edited by @mtoneva1 : paper:

0

13

37

0

4

18

@pliang279

Paul Liang

3 years

We're organizing the 3rd workshop on multimodal AI @NAACLHLT ! We welcome submission on all areas and applications of multimodal language learning. Deadline: March 15 2021 with fantastic keynote speakers Kristen Grauman, @aninddey , @emilymprovost webpage:

Tweet media one

Tweet media two

Tweet media three

1

5

19

@pliang279

Paul Liang

2 years

[11877 Advanced Topics in Multimodal ML] In week 4’s session, the class discussed recent trends of large-scale pretrained language and multimodal models, and the overall risks and opportunities offered by the pretraining paradigm. Notes here:

Tweet media one

Tweet media two

1

5

17

@pliang279

Paul Liang

5 years

My advisor LP Morency @LTIatCMU @mldcmu @SCSatCMU has done fantastic work using multimodal human behaviors to detect depression, schizophrenia, PTSD, and those at risk of suicide. Please vote for LP to get into the #SXSW2020 panel!

Tweet card media

The Next Step in AI: Multimodal Perception | Louis-Philippe Morency |...

Human face-to-face communication is a little like a dance: participants continuously adjust their behaviors based on their interlocutor’s speech, gestures an...

www.youtube.com

0

2

17

@pliang279

Paul Liang

4 years

All the graduate applicant support programs in one thread! Get your application together early and receive feedback from current CS PhD students. I highly encourage everyone to apply - esp students from diverse backgrounds and educational paths.

@andrewkuznet

Andrew Kuznetsov

4 years

This year, CS PhD applications are different. There are many more 'graduate application support programs' for applicants to get informal feedback (on statements, etc) from current PhD students before formally applying. It's an awesome resource! I've link some below:

4

91

286

0

9

16

@pliang279

Paul Liang

4 years

#NeurIPS deadline extension to 3 June!

Tweet media one

2

1

16

@pliang279

Paul Liang

2 years

@rsalakhu @lpmorency @LTIatCMU @mldcmu HighMMT standardizes input modalities into sequences and uses modality-specific embedding layers to capture unique information. The rest of the model learns modality and task-agnostic representations through shared unimodal and multimodal layers trained via multitask learning.

Tweet media one

1

5

17

@pliang279

Paul Liang

2 years

Excited to share our new benchmark, PACS: an audiovisual question-answering dataset for physical commonsense reasoning and new models at #ECCV2022 ! Paper: Code/Dataset: w Samuel Yu @peter_yh_wu @rsalakhu @lpmorency

1

3

17

@pliang279

Paul Liang

5 years

2 great papers at #ICML2019 study this theoretical and empirically: Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations (), and Disentangling Disentanglement in Variational Autoencoders ()

Tweet card media

Disentangling Disentanglement in Variational Autoencoders

We develop a generalisation of disentanglement in VAEs---decomposition of the latent representation---characterising it as the fulfilment of two factors: a) the latent encodings of the data having...

1

6

15

@pliang279

Paul Liang

1 year

Check out our latest work at #ACL2023NLP on improving compositionality in vision-language models, through aligning not just entities but also relations between words and image regions see 🧵 by @khoomeik

Tweet card media

Cross-modal Attention Congruence Regularization for...

Despite recent progress towards scaling up multimodal vision-language models, these models are still known to struggle on compositional generalization benchmarks such as Winoground. We find that a...

@khoomeik

Rohan Pandey (e/acc)

1 year

Complex multimodal reasoning requires not only entities to be matched between an image and text, but also their relations. Check out our work at #ACL2023 Poster Session 2 (Monday 2pm) where we propose a regularization objective that encourages cross-modal relation alignment.

Tweet media one

3

18

64

0

4

16

@pliang279

Paul Liang

5 years

2. Words Can Shift: Dynamically Adjusting Word Representations Using Nonverbal Behaviors @ 2pm, South Pacific 2 with Yansen Wang, @YingShen_ys , @ZhunLiu3 , Amir Zadeh, LP Morency @mldcmu @LTIatCMU @SCSatCMU #AAAI2019 #AAAI19

Tweet media one

Tweet media two

Tweet media three

1

4

16