Xingyu Lin @Xingyu2017 Twitter profile

Pinned Tweet

Xingyu Lin

@Xingyu2017

7 months

What state representation should robots have? 🤖 I’m thrilled to present an Any-point Trajectory Model (ATM), which models physical motions from videos without additional assumptions and shows significant positive transfer from cross-embodiment human and robot videos! 🧵👇

9

105

538

Last Seen Profiles

@LaurenLenox

@FMV_VSE

@Lusa_noticias

@krr150ssr

@cenarockswwe

@BABGSacha

@poptelevisao

@yuarikka

@trolo_xy

@uftpkgXoG30ZmUt

@tangmo0608

@JimHam18

@cindyDev_

@EvgheniRoman

@sajid_abid

@LydiaReddic

@DigiBankSummit

@hexcidelol

@1EOIL5vKxSc6gvy

@F3Crypt

@iw9HiUMfGQhR4pl

@AceStapp

@Krokant_riyadh

@butthisisshort

@hanyan_0906tm

@bokeplokalmalam

@Yilmaz_Ozdill

@XiaoHu1

@ShadowDarkknig1

@christinaluvsts

@carmodyp18

@BolandsWexford

@SapphireD3564

@dinjoyer

@sotwecom

@bokeplokalmalam

Xingyu Lin

@Xingyu2017

1 year

Excited to share my first paper at UC Berkeley! We identify key bottlenecks in learning from a pre-trained visual representation and show generalization to novel objects from only three instances.🧵⬇️ Website:

2

42

225

Xingyu Lin

@Xingyu2017

4 years

Excited to share our #CoRL2020 paper where we present SoftGym, the first benchmark for deformable object manipulation and show that the tasks present great challenges for RL. Project page: w/ @YufeiWang15 Jake Olkin, @davheld . @corl_conf

5

29

199

Xingyu Lin

@Xingyu2017

3 years

For smoothing crumpled cloths, we found that a mesh based graph neural network achieves better performance, generalization to novel shapes and materials, and makes it easy to transfer to the real world! To appear at #CORL2021 @CMU_Robotics Website:

1

15

121

Xingyu Lin

@Xingyu2017

2 years

Object-centric representations and hierarchical reasoning are key to generalization. How can we manipulate deformables, where “objectness” changes over time? Our method finds a way and solves challenging real-world dough manipulation tasks! #CoRL2022

2

25

116

Xingyu Lin

@Xingyu2017

2 years

Robotic manipulation of deformable objects like dough requires long-horizon reasoning over the use of different tools. Our method DiffSkill utilizes a differentiable simulator to learn and compose skills for these challenging tasks. #ICLR2022 Website:

2

16

94

Xingyu Lin

@Xingyu2017

10 months

I’ll be in Atlanta for #CoRL2023 and will present our recent works on SpawnNet ( #NeuRL4RM workshop) and GELLO ( #TGR ) tomorrow. Also excited to share that I’m on the job market, looking for tenure-track positions in AI and robotics. Would love to chat about potential fit!

Xingyu Lin

@Xingyu2017

1 year

Excited to share my first paper at UC Berkeley! We identify key bottlenecks in learning from a pre-trained visual representation and show generalization to novel objects from only three instances.🧵⬇️ Website:

2

42

225

0

6

51

Xingyu Lin

@Xingyu2017

4 months

2/2 papers accepted at #RSS2024 🥳. Huge congratulations to my incredible collaborators! Check out our work on trajectory modeling from videos: and humanoid benchmark for whole-body control:

0

4

41

Xingyu Lin

@Xingyu2017

4 months

Predicting future point trajectory can serve as a "language" for actions, bridging different embodiments. It's exciting to see new work advancing in this direction! If you are interested, also also check out our prior ATM paper:

Homanga Bharadhwaj

@mangahomanga

4 months

Track2Act: Our latest on training goal-conditioned policies for diverse manipulation in the real-world. We train a model for embodiment-agnostic point track prediction from web videos combined with embodiment-specific residual policy learning 1/n

2

29

123

1

6

32

Xingyu Lin

@Xingyu2017

4 months

I will be at ICRA next week to present SpawnNet! Happy to chat about pre-training from videos or robot learning in general :)

Xingyu Lin

@Xingyu2017

1 year

Excited to share my first paper at UC Berkeley! We identify key bottlenecks in learning from a pre-trained visual representation and show generalization to novel objects from only three instances.🧵⬇️ Website:

2

42

225

0

1

29

Xingyu Lin

@Xingyu2017

4 years

How should we combine multiple auxiliary tasks to accelerate RL? Check out our #NeurIPS2019 paper that provides a principled method in this direction: Paper: Code: @davheld @HarjatinS

0

10

28

Xingyu Lin

@Xingyu2017

6 months

Introducing a fully open-sourced simulation benchmark with challenging whole-body control tasks and whole-body sensing!

Carlo Sferrazza

@carlo_sferrazza

6 months

Humanoids 🤖 will do anything humans can do. But are state-of-the-art algorithms up to the challenge? Introducing HumanoidBench, the first-of-its-kind simulated humanoid benchmark with 27 distinct whole-body tasks requiring intricate long-horizon planning and coordination. 🧵👇

8

91

332

0

2

27

Xingyu Lin

@Xingyu2017

7 months

@n_karaev @chrirupp @CarlDoersch 5/5 Work done with great collaborators @ChuanWen15 , @johnrso_ , Kai Chen, Qi Dou, Yang Gao, and @pabbeel ! Project website: Paper:

Any-point Trajectory Modeling for Policy Learning

Learning from demonstration is a powerful method for teaching robots new skills, and having more demonstration data often improves policy learning. However, the high cost of collecting...

arxiv.org

0

3

23

Xingyu Lin

@Xingyu2017

10 months

Code and hardware instructions released!

Philipp Wu

@philippswu

10 months

We have released the hardware files and instructions for GELLO 🦾! See the updated website . Here's a video of me assembling it start to finish in 30 min! GELLO is also at CoRL this week, see 👇 for details

7

22

110

0

2

22

Xingyu Lin

@Xingyu2017

1 year

Introducing GELLO: Your gateway to intuitive robot arm teleoperation with a cost of under $300! 🚀

Philipp Wu

@philippswu

1 year

🎉Excited to share a fun little hardware project we’ve been working on. GELLO is an intuitive and low cost teleoperation device for robot arms that costs less than $300. We've seen the importance of data quality in imitation learning. Our goal is to make this more accessible 1/n

26

110

688

0

5

21

Xingyu Lin

@Xingyu2017

7 months

4/5 Our work is enabled by recent advances in video tracking. We build on top of the great works from CoTracker ( @n_karaev @chrirupp ) and Tracking-Any-Point by @CarlDoersch et al.

1

2

19

Xingyu Lin

@Xingyu2017

2 years

Self-occlusion is a challenging problem in cloth manipulation. Come and check out our recent paper presented at #RSS2022 tomorrow. Lead by the wonderful @ZixuanHuang15

Zixuan Huang

@ZixuanHuang15

2 years

How can we enable a robot to explicitly reason about occlusions for better cloth manipulation? Check out our #RSS2022 paper which proposes a self-supervised test-time finetuning method for reconstructing crumpled clothes. w/ @davheld , @xingyu2017 Website:

1

7

28

0

2

16

Xingyu Lin

@Xingyu2017

7 months

1/5 Our goal is to improve policy learning from video data, a rich and scalable source. Since videos lack explicit actions, we focus on learning to predict the future trajectories of any set of particles based on their initial 2D positions, circumventing the need for actions

1

0

15

Xingyu Lin

@Xingyu2017

10 months

Generating diverse tasks/scenes is always a time-consuming part when building simulation environments. Very excited to see generative models being used to scale up the diversity in simulation!

Zhou Xian

@zhou_xian_

10 months

Can GPTs generate infinite and diverse data for robotics? Introducing RoboGen, a generative robotic agent that keeps proposing new tasks, creating corresponding environments and acquiring novel skills autonomously! code: 👇🧵 (better with audio)

10

84

312

0

4

15

Xingyu Lin

@Xingyu2017

7 months

3/5 By modeling the low-level particle trajectories, we find significant positive transfer from videos of humans or from a different robot! Our current model is trained from relatively in-domain videos. Stay tuned for developments on a more generalized model!

1

0

14

Xingyu Lin

@Xingyu2017

4 months

#ICRA2024 Will present at the poster session at 10:30 AM today and oral session at 4:30 pm! Exhibition hall, booth 0304.

Xingyu Lin

@Xingyu2017

1 year

Excited to share my first paper at UC Berkeley! We identify key bottlenecks in learning from a pre-trained visual representation and show generalization to novel objects from only three instances.🧵⬇️ Website:

2

42

225

0

12

Xingyu Lin

@Xingyu2017

2 years

Fruit Ninja, but for a robot! A super fun project led by the amazing @Zhenjia_Xu

Shuran Song

@SongShuran

2 years

The Internet is too fast, I’m still crafting my catchy twits, and word is already out😂 Well then, now you have it: RoboNinja🥷: Learning an Adaptive Cutting Policy for Multi-Material Objects 🧵👇 for a few interesting details you might have missed

4

51

364

0

1

11

Xingyu Lin

@Xingyu2017

7 months

2/5 Once the trajectory model is trained, we learn trajectory-guided policies. We simply look at the trajectories of points from a fixed grid. We do not assume any calibration and our model utilizes cameras of different viewpoints.

1

0

10

Xingyu Lin

@Xingyu2017

1 year

Visual pre-training on internet-scale vision datasets has the potential to enable generalizable manipulation. But tasks in prior works have limited variations. A recent paper from @ncklashansen shows that simple data augmentation is competitive to the SOTA visual pre-training

1

0

9

Xingyu Lin

@Xingyu2017

2 years

Code has been released!

GitHub - Xingyu-Lin/PASTA: Official implementation of the DiffSkill and PASTA algorithms for...

Official implementation of the DiffSkill and PASTA algorithms for long-horizon, skill-based deformable object manipulation. - Xingyu-Lin/PASTA

github.com

0

1

7

Xingyu Lin

@Xingyu2017

1 year

@ncklashansen This is joint work with @johnrso_ , Sashwat, @fangchenliu_ and @pabbeel Paper: Code:

SpawnNet: Learning Generalizable Visuomotor Skills from...

The existing internet-scale image and video datasets cover a wide range of everyday objects and tasks, bringing the potential of learning policies that generalize in diverse scenarios. Prior works...

arxiv.org

0

1

7

Xingyu Lin

@Xingyu2017

6 months

The difficulty of a manipulation task is largely defined by the amount of task variations the robot needs to handle, such as object geometries and poses. Glad to see new robot demos with more diverse objects!

1X

@1x_tech

6 months

1X’s mission is to create an abundant supply of physical labor through androids that work alongside humans. We're excited to share our latest progress on teaching EVEs general-purpose skills. The following is all autonomous, all 1X speed, all controlled with a single set of

69

182

1K

0

7

Xingyu Lin

@Xingyu2017

1 year

@ncklashansen In this project, we build a set of challenging tasks where policies are trained on a few instances and evaluated on held-out, novel objects from the same category.

1

0

6

Xingyu Lin

@Xingyu2017

2 years

Part 2/3 of our dough manipulation projects at CMU!

Carl Qi

@carl_qi98

2 years

Check out our work on learning closed-loop dough manipulation: we use a differentiable reset module to avoid local optima from gradient-based trajectory optimization! #RAL2022 #IROS2022 w/ Xingyu Lin @Xingyu2017 , and David Held @davheld .

2

7

37

0

1

6

Xingyu Lin

@Xingyu2017

1 year

@ncklashansen We propose a novel architecture for learning from pre-trained networks that address the key bottleneck of a frozen pre-trained representation. Our method is very simple but shows significant and consistent improvements over prior works!

1

0

6

Xingyu Lin

@Xingyu2017

2 years

3/7 This latent set representation has two benefits: one is to effectively model the change in the number of components during an episode, e.g. a piece of dough being cut into two. The second benefit is to enable compositional generalization to more components at test time.

1

2

5

Xingyu Lin

@Xingyu2017

2 years

This is a joint work w/ Carl Qi , Yunchu Zhang, Zhiao Huang, Yunchu Zhang, Katerina Fragkiadaki, Yunzhu Li , Gan Chuang and David Held. @carl_qi98 @huang_zhiao @YunzhuLiYZ @gan_chuang @davheld

1

5

Xingyu Lin

@Xingyu2017

4 years

Join us for the talk and interactive session on Mon, 16th Nov - 11:50 AM PST!

Yufei Wang

@YufeiWang15

4 years

Check out our #CoRL2020 method, ROLL, which uses object reasoning and occlusion reasoning to learn a self-supervised reward for visual RL to learn various manipulation skills! w/ @gauthamnarayn @Xingyu2017 @okorn_brian @davheld @corl_conf paper:

1

0

8

0

5

Xingyu Lin

@Xingyu2017

3 years

@CMU_Robotics w/ @YufeiWang15 * , @ZixuanHuang15 and @davheld Come visit our poster on Thursday in Poster Sesssion VI!

1

0

4

Xingyu Lin

@Xingyu2017

7 months

@juanstoppa Will release the code soon. Stay tuned!

0

4

Xingyu Lin

@Xingyu2017

3 years

Code released here:

GitHub - Xingyu-Lin/VCD: Code for the paper Learning Visible Connectivity Dynamics for Cloth...

Code for the paper Learning Visible Connectivity Dynamics for Cloth Smoothing - Xingyu-Lin/VCD

github.com

0

3

Xingyu Lin

@Xingyu2017

4 months

@chris_j_paxton Thank you Chris! Indeed, I am very excited to see the further development of this work in even more generalization!

0

3

Xingyu Lin

@Xingyu2017

2 years

This is joint work with @huang_zhiao , @YunzhuLiYZ ,Josh Tenenbaum, @davheld , @gan_chuang . We will present our poster virtually at #ICLR2022 on Thursday at 1:30 pm EST. Come and chat with us!

1

3

Xingyu Lin

@Xingyu2017

7 months

@koval_alvi On GPU the model runs in real time. But running on CPU can be much slower.

0

3

Xingyu Lin

@Xingyu2017

4 years

What a way to comfort a Federer fan after Wimbledon 2019😂

AK

@_akhaliq

4 years

Vid2Player: Controllable Video Sprites that Behave and Appear like Professional Tennis Players pdf: abs: project page:

39

552

2K

0

3

Xingyu Lin

@Xingyu2017

10 months

@haosu_twitr Congrats!!!

1

0

1

Xingyu Lin

@Xingyu2017

4 years

Join us for the talk and interactive session on Tue, 17th Nov - 11:10 AM PST! Summary Video (5 mins):

CoRL 2020, Spotlight Talk 91: SoftGym: Benchmarking Deep Reinforcem...

"**SoftGym: Benchmarking Deep Reinforcement Learning for Deformable Object Manipulation** Xingyu Lin (Carnegie Mellon University)*; Yufei Wang (Carnegie Mell...

www.youtube.com

0

3

Xingyu Lin

@Xingyu2017

2 years

2/7 To solve long-horizon tasks, we reason over a spatial and temporal abstraction. We obtain a spatial abstraction by clustering the points into different components based on their proximity in space. We encode each component to a latent representation to obtain a latent set.

1

2

Xingyu Lin

@Xingyu2017

2 years

5/7 Given an observed and target point cloud, we encode them into latent sets and then plan latent subgoals by optimizing a combination of the feasibility scores and the cost to chain the skills.

1

2

Xingyu Lin

@Xingyu2017

2 years

@Zyc199539Chu

0

2

Xingyu Lin

@Xingyu2017

8 months

@stepjamUK The layout looks fantastic!

0

2

Xingyu Lin

@Xingyu2017

2 years

7/7 Our method outperforms all the baselines in simulation and excels at tasks (CutRearrangeSpread, CRS-Twice) where there are more components and planning steps at test time.

1

2

Xingyu Lin

@Xingyu2017

4 years

Code link:

GitHub - Xingyu-Lin/softgym: SoftGym is a set of benchmark environments for deformable object...

SoftGym is a set of benchmark environments for deformable object manipulation. - Xingyu-Lin/softgym

github.com

1

0

2

Xingyu Lin

@Xingyu2017

6 months

@sehoonha Congrats!!

0

2

Xingyu Lin

@Xingyu2017

2 years

1/7 Given some skill demonstration trajectories (obtained by differentiable trajectory optimization), we first learn goal-conditioned policies from these trajectories via BC+HER. Each skill uses one tool to manipulate the dough and needs to be chained to solve multi-stage tasks.

1

2

Xingyu Lin

@Xingyu2017

2 years

The resulting method is named PASTA: PlAnning with Spatial-Temporal Abstraction. It can reason over long horizon tasks. We transfer the planner to the real world without any fine-tuning.

1

2

1

Xingyu Lin

@Xingyu2017

2 years

@GuanyaShi @CarnegieMellon @CMU_Robotics @SCSatCMU @yisongyue Congratulations!

0

1

Xingyu Lin

@Xingyu2017

1 year

@davheld @CMU_Robotics Congratulations, Dave!

0

1

Xingyu Lin

@Xingyu2017

5 years

@NotionHQ @Quelklef +1

0

1

Xingyu Lin

@Xingyu2017

2 years

4/7 To chain skills, we also learn temporal abstraction modules: one feasibility predictor per skill (predicts the likelihood of reaching one state from another using the learned skill) and a cost function. Both modules take the set representation of the observation and the goal.

1

Xingyu Lin

@Xingyu2017

4 months

@servo_boyd You are right. Given the limited video data we are training, I do not expect the model to generalize across large viewpoint variation. Minor camera jittering might be fine.

0

1

Xingyu Lin

@Xingyu2017

1 year

@m0hitsharma Hey Mohit, thanks for the pointer! sim2real transfer is another benefit of using pre-trained networks, while our paper focuses more on categorical generalization.

0

1

Xingyu Lin

@Xingyu2017

1 year

@haosu_twitr @andyzeng_ @QinYuzhe @GeorgiaChal @JoeLilKim @ShanLuoRobotics Thanks to all organizers for the great workshop! A lot of inspiring discussions.

0

1

Xingyu Lin

@Xingyu2017

5 years

@NotionHQ @cindywu @skhanyz +1

0

1

Xingyu Lin

@Xingyu2017

5 years

@NotionHQ @beanstalkpark +1

0

1

Xingyu Lin

@Xingyu2017

4 months

@shahdhruv_ Congrats, Dhruv!

0

1

Xingyu Lin

@Xingyu2017

2 years

1/5 It is not easy to manually define skills such as spreading or gathering dough using tools. As such, we run gradient-based trajectory optimization in a differentiable simulator to solve for trajectories that can reach short-horizon goals.

1

0

1