Stephen James @stepjamUK Twitter profile

Last Seen Profiles

@MiraclePel33755

@xtrops3

@ge_tig

@badandbitchy

@jsuhowsky

@BoooN_139

@Calamity_Corp

@siro__xo

@eva_moreda

@YmnyHrrr39

@BLACKEDSISSY19

@CujosApparel

@itigo_miuku

@MrAllen45

@DMarhencke

@HarryOGK_115

@gyucart_s

@bokeplokalmalam

@AorRabitt

@adasbowgun

@seishow_kame

@bokeplokalmalam

@bartosz64982776

@jandakembangstw

@ppxxloy

@stw46

@Susofn_

@time_simpler

@moppi0x5D77

@Coach_Kennedy

@aviation07101

@ume__chii

@stw_pdg

@roypete23694563

@chapanino

Stephen James

@stepjamUK

7 months

Dyson Robot Learning Lab is Hiring full-timers and interns! 🤖 1x Research Scientist: 1x Data Engineer (Data Collection & ML Training): 3x PhD Internship: Come join our lab of 12; located in London, UK 🇬🇧

7

60

327

Stephen James

@stepjamUK

1 year

We have a Fall PhD Internship opening at the Dyson Robot Learning Lab in London! 🇬🇧 Come and join our world-class team of robot learning gurus for 3-6 months! Apply via form:

6

58

282

Stephen James

@stepjamUK

2 years

New paper! Not all reinforcement learning problems are suited for a Gaussian policy parameterization!🤯 Plan to use 3D rotation or 6D pose as part of an action space? Consider the Bingham distribution! 🧵1/5 Paper: Code: w/ @pabbeel

4

57

274

Stephen James

@stepjamUK

2 years

Following the success of Masked AutoEncoders (MAE), we all knew it was coming.... VideoMAE. The death of contrastive learning for video representation learning? I think so. Props to the rapid pace of the authors.👏 MAE has only been out for ~3 months🤯

4

40

255

Stephen James

@stepjamUK

2 years

A year ago, I thought it obvious that meta-RL had a role to play in robotics... Now I’m no longer convinced! Our large-scale study shows multi-task pretraining followed by fine-tuning on novel tasks, performs >= meta-RL! Lead: @ZhaoMandi w/ @pabbeel

7

40

248

Stephen James

@stepjamUK

6 years

Sim-to-Real via Sim-to-Sim: we learn a generator that translates real-world images to a canonical simulation version to learn robot grasping with no real-world data! w/ collaborators from @GoogleAI , @Theteamatx , @DeepMindAI

5

67

231

Stephen James

@stepjamUK

5 years

We are happy to announce PyRep - a toolkit for rapid robot learning research! PyRep is a modification to V-REP that is ~10,000x faster vs the previous remote client approach! Work with @coppeliaRobotic & @AjdDavison . Report: Code:

4

54

206

Stephen James

@stepjamUK

2 years

Coarse-to-fine Q-attention has been selected for an oral at #CVPR2022 !🙀 The first *general* 6DoF RL-based manipulation algorithm that can train in the real world in minutes (not days/months). High-res images, and no shaped rewards!

4

29

193

Stephen James

@stepjamUK

27 days

🚨Important update from our Robot Learning Lab in London. Following recent news, we’re moving on after a wonderful 2 years… Today, we unveil 4 big pieces of research from our incredible team. Check out the compilation video and thread below to see our final work! 📽️👇

5

38

183

Stephen James

@stepjamUK

2 months

Render and Diffuse (R&D) uses joint observation-action representation to learn low-level robot actions using a learnt diffusion process that iteratively updates virtual renders of robot actions leading to big sample-efficiency gains 💪 #RSS2024 Dyson RLL

0

27

180

Stephen James

@stepjamUK

2 years

Our new Sim2Real 6D Object Pose Estimation work! 1) Train pose estimation model on sim data. 2) Use model to generate poses on unlabelled real data. 3) Auto filter generated poses and update model. 4) Repeat 2-4. Result: SOTA performance + robot demo! 🤖

4

37

178

Stephen James

@stepjamUK

5 months

Hierarchical diffusion policy is another step along the journey of making hierarchical next-best pose agents more capable, through introduction of a kinematically-aware low-level diffusion planner.🤖 New work from the Dyson Robot Learning Lab. CVPR 2024

1

22

137

Stephen James

@stepjamUK

2 years

@corl_conf Oral! Self-supervised in-the-wild video pre-training for real-world robotic tasks!🤖 Our 307M parameter vision transformer outperforms CLIP, ImageNet pre-training, and training from scratch! 📜 🌐 🧠

1

25

125

Stephen James

@stepjamUK

2 years

Announcing the 1st "Workshop on Pre-training Robot Learning" at @corl_conf , Dec 15. Fantastic lineup of speakers: Jitendra Malik, Chelsea Finn, Joseph Lim, Kristen Graumen, Abhinav Gupta, Raia Hadsell. Submit your 4-page extended abstract by September 28.

3

33

116

Stephen James

@stepjamUK

2 years

New work! We envision coarse-to-fine Q-attention as a tree that can be expanded and used to accumulate value estimates across the top-k voxels at each Q-attention depth. Allows for more robust sparse-reward, vision-based manipulation! 🧵👇 w/ @pabbeel

2

23

114

Stephen James

@stepjamUK

3 years

2 life updates! (1) Yesterday I passed my PhD viva! Thank you to my examiners Jens Kober and @Petar_Kormushev , and of course, my supervisor @AjdDavison ! (2) In June I begin a postdoc at @pabbeel 's group at UC Berkeley! Looking forward to continuing making robots see and do! 🤖

19

0

114

Stephen James

@stepjamUK

5 years

We are thrilled to announce RLBench: an ambitious large-scale benchmark and learning environment for vision-guided manipulation with 100 unique, hand-design tasks! Paper: Video: w/ @StephenLJames , Z. Ma, D. Arrojo, @AjdDavison

RLBench: The Robot Learning Benchmark

Authors: Stephen James, Zicong Ma, David Rovick Arrojo, Andrew J. Davison.Dyson Robotics Lab, Imperial College London.Music Credit:––––––––––––––––––––––––––...

www.youtube.com

2

40

114

Stephen James

@stepjamUK

3 years

'basketball_in_hoop'; one of many new tasks joining the #RLBench family of 100+ tasks in V1.2. Coming early November! 🤖 RLBench is still the hardest manipulation sim-benchmark to date due to its large-scale focus on vision, sparse rewards, and multi-stage tasks.

0

18

108

Stephen James

@stepjamUK

2 years

New! Learned Path Ranking (LPR) takes a Q-attention next-best pose, and learns to rank a set of goal-reaching paths generated by path planning, Bezier curve sampling, and a learned policy. We can now accomplish more RLBench and real-world tasks! w/ @pabbeel

2

18

92

Stephen James

@stepjamUK

3 years

New work! We generate 3D shape + segmentation from depth, probabilistically sampling occluded region proposals; giving robots the means to intuitively reason about partially observed scenes, allowing grasps without causing stacks to fall! Lead: @LandgrafZoe

2

6

91

Stephen James

@stepjamUK

3 years

Attention-driven Robot Manipulation (ARM)🦾 is our new learning algorithm that can do RLBench tasks, while other baselines fail. Secret sauce is Q-attention🔎, which crops around interesting pixels before giving to an actor-critic method. w/ @AjdDavison

1

13

83

Stephen James

@stepjamUK

5 months

High-level agent predicts next-best pose (e.g. C2F-ARM/PerAct/etc), which is then fed into a kinematically aware diffusion policy. Pros of this over C2F/PerAct is that it doesn't rely on motion planning; here we are essentially "learning" the motion planning component. #CVPR2024

Stephen James

@stepjamUK

5 months

Hierarchical diffusion policy is another step along the journey of making hierarchical next-best pose agents more capable, through introduction of a kinematically-aware low-level diffusion planner.🤖 New work from the Dyson Robot Learning Lab. CVPR 2024

1

22

137

1

16

82

Stephen James

@stepjamUK

2 months

🚨End-effector Redundancy (ER) Action Space for Robot Manipulation! It has the *sample efficiency* of task-space (end-effector) control, but the versatility of joint control! Now one of our de-facto action modes we use in the Dyson Robot Learning Lab!

Pietro Mazzaglia @ ICML

@pietromazzaglia

2 months

Novel action spaces leveraging redundancy in 7 DoF arms enable efficient & precise learning in robotic manipulation 🤖 Current action spaces often fall short in human environments, where solving complex tasks requires avoiding obstacles & reaching confined spaces. 1/n

1

15

83

2

15

82

Stephen James

@stepjamUK

9 months

My proudest PhD work was action-value next-best pose (Q-attention). Really impressed to see so much cool work use this! PerAct, RVT, Act3D. What next? Fun fact: it was first rejected, then resubmitted unchanged and got an oral at CVPR 22. Don't give up on work you believe in.

Eric Jang

@ericjang11

9 months

great work! It is not well known outside of the robotic visual manipulation community but these sort of action-value maps tend to be the most performant when it comes to high success rates for object manipulation.

2

12

81

1

5

70

Stephen James

@stepjamUK

2 years

StereoPose! Our new stereo RGB framework for category-level object pose estimation that works incredibly well for transparent objects! The magic sauce? Back-view normalized object coordinate space (NOCS)! More below ⬇️. 📜 Lead: Kai Chen, @CUHKofficial

6

19

65

Stephen James

@stepjamUK

2 years

We are now in full hiring mode! Research Scientist in Robot Learning: Simulation Engineer (Unity Game Dev): Fancy moving from the games industry to robotics? This SimEng role could be for you! Robot/ML Eng applications closed!

Research Scientist, Dyson Robot Learning Lab, London

About Us At Dyson, our goal is to build a world-leading robotics team focused on developing advanced domestic robots to go into real homes and help real people. Our multidisciplinary team is already...

docs.google.com

3

14

66

Stephen James

@stepjamUK

4 years

Very excited to be invited to talk at the 6th International Workshop on Recovering 6D Object Pose at #ECCV2020 . I'll be talking about multi-object reasoning, end-to-end #manipulation , and #sim2real . See you Sunday at 10.10 (UTC+1).

0

9

62

Stephen James

@stepjamUK

6 years

#RobotriX : a photorealistic indoor dataset for robotic vision. Paper:

2

23

63

Stephen James

@stepjamUK

4 years

MoreFusion #CVPR2020 is our new real-time and incremental pose estimation system (achieves SOTA) that builds an object-level map describing the full geometry of objects in scene; allows precise pick and place of cluttered objects. Work led by @wkentaro_ .

0

9

58

Stephen James

@stepjamUK

2 years

New! Patch-based Object-centric Video Transformer. We use object-centric information (bounding boxes) as a compressed representation for videos, giving improved computational efficiency on long-horizon video prediction. Lead: Wilson Yan w/ Ryo O, @pabbeel

2

11

58

Stephen James

@stepjamUK

2 years

PhD internship applications are now OPEN from spring 2023 onwards at the Dyson Robot Learning Lab in London. They can be undertaken at any time during the year! Apply through this form⬇️

0

8

55

Stephen James

@stepjamUK

5 years

Today at #CVPR2019 we are presenting Sim-to-Real via Sim-to-Sim: Data-efficient Robotic Grasping via Randomized-to-Canonical Adaptation Networks (RCAN). Come and see us at Poster 204!

0

8

54

Stephen James

@stepjamUK

2 years

Today at 3pm (BST), I'll be talking about sample-efficient robot learning at ETH Zurich. Topics include Q-attention, model-based RL, and pre-training for robot control. Come join! No registration required; link to zoom can be found here:

1

3

52

Stephen James

@stepjamUK

6 years

In our oral at #CoRL2018 tomorrow, I will present Task-Embedded Control Networks, which employ ideas from metric learning in order to create a task embedding that can be used by a robot to learn new tasks from one or more demonstrations. #robotics #MachineLearning #AI #phdlife

2

12

52

Stephen James

@stepjamUK

5 years

If you haven't already, please check out what I worked during my @Theteamatx internship! Randomized-to-Canonical Adaptation Network (RCAN) is a real2sim image translator trained with domain randomization and achieves SoTA performance on robotic grasping!

Sim-to-Real via Sim-to-Sim: Data-efficient Robotic Grasping via...

Real world data, especially in the domain of robotics, is notoriously costly to collect. One way to circumvent this can be to leverage the power of simulation to produce large amounts of labelled...

arxiv.org

Google DeepMind

@GoogleDeepMind

5 years

Together with @Theteamatx and @GoogleAI , we have recently proposed the Randomized-to-Canonical Adaptation Network (RCAN): a real2sim image translator trained with domain randomization. It achieves SoTA performance on robotic grasping with no real data.

11

110

371

2

9

50

Stephen James

@stepjamUK

1 year

Woohoo! Super excited about this new work! Multi-View Masked World Models (MV-MWM) gives super robust end-to-end manipulation with RGB input from a *hand-held camera*! Looks like our camera operator had a few too many beers!🍻 @younggyoseo thread below!

Younggyo Seo

@younggyoseo

1 year

Excited to share Multi-View Masked World Models (MV-MWM) that learns multi-view representations and a world model for viewpoint-robust control! MV-MWM enables zero-shot sim2real transfer with hand-held cameras, only using pixel observations 🚀

1

42

204

0

3

45

Stephen James

@stepjamUK

2 years

Our mission? Deploy advanced robots in household environments. I'll be Moving into my full time Dyson role starting August! Big shoutout to my advisors ( @pabbeel and @AjdDavison ) who have inspired me over the years! Full Dyson Robotics video reveal:

Dyson gives glimpse of secret robot prototypes which appear to carry...

Need a helping hand around the home?Today, at the International Conference on Robotics and Automation in Philadelphia, Dyson gives a glimpse into the future ...

www.youtube.com

3

5

42

Stephen James

@stepjamUK

5 years

#PyRep update! Say hello to mobile bases! Here's a peak at LoCoBot from @facebookai & @CMU_Robotics . Update also includes TurtleBot and @KUKAGlobal YouBot. PyRep: Paper:

0

10

40

Stephen James

@stepjamUK

1 year

Video-language models (VLMs) are often used as a means to solve sparse rewarded tasks. The issue is, they kinda suck giving meaningful rewards 💩. In this work, we question whether VLMs are perhaps best-suited as a *pretraining* signal for RL. LAMP💡:

Language Reward Modulation for Pretraining Reinforcement Learning

Using learned reward functions (LRFs) as a means to solve sparse-reward reinforcement learning (RL) tasks has yielded some steady progress in task-complexity through the years. In this work, we...

arxiv.org

Ademi Adeniji

@AdemiAdeniji

1 year

Excited to share LAnguage Reward Modulation for Pretraining Reinforcement Learning! LAMP💡pretrains a language-conditioned agent without human supervision using VLM rewards and unsupervised reinforcement learning w/ @amberxie_ @carlo_sferrazza @younggyoseo @stepjamUK @pabbeel

2

23

109

0

2

39

Stephen James

@stepjamUK

5 years

Insightful paper! They study the effect of different action spaces in deep RL. After evaluating multiple action spaces across three #manipulation tasks, they conclude that variable impedance control in end-effector space is the winner!

1

9

36

Stephen James

@stepjamUK

2 years

New work! Can Masked Autoencoders (MAE) be effective for visual model-based RL? Yes! ✅ Our Masked World Model (MWM) decouples visual representations and dynamics by training a latent dynamics model on features from a pre-trained MAE.

Masked World Models for Visual Control

Visual model-based reinforcement learning (RL) has the potential to enable sample-efficient robot learning from visual observations. Yet the current approaches typically train a single model...

arxiv.org

Younggyo Seo

@younggyoseo

2 years

Excited to share Masked World Models for Visual Control! Inspired by MAE and World Models, we train an autoencoder with convolutional feature masking and reward prediction, then train a dynamics model in the latent space of the autoencoder. Thread 🧵

5

61

271

2

6

34

Stephen James

@stepjamUK

5 years

Table-top instance grasping of objects using no real world grasping data! Method consists of a shape prediction model that learns a domain-invariant 3D point cloud representation of objects which is consequently used for grasping (via a critic network). #Robotics

2

7

33

Stephen James

@stepjamUK

5 years

Over the next few weeks, I'll periodically post various tasks/features from #RLBench . First up, a shape sorting task! This one would be a great task to try for a sim-to-real project! Both vision and proprioceptive feedback needed for this one.

2

4

34

Stephen James

@stepjamUK

2 years

Fantastic news from @coppeliaRobotic : it now supports MuJoCo (in addition to the already supported Bullet, ODE, Newton, Vortex). A reminder that RLBench is built around CoppeliaSim; no better tool out there for fast robot learning research iteration 🙂

Coppelia Robotics

@coppeliaRobotic

2 years

Happy to announce a new release of the #RoboticsSimulator #CoppeliaSim . Some features: - #MuJoCo now also supported with dyn. content that can be modified on-the-fly. Bringing soft bodies, cables, etc. to CoppeliaSim - #python also supported for embedded scripts - ROS2 Humble

2

13

93

0

1

34

Stephen James

@stepjamUK

27 days

As we explore new opportunities and the future of this talented group, we’re grateful for all the support. Feel free to reach out—our DMs are open! @mohito1905 @younggyoseo @iainhaughton @nc__dev @chrysalis_ai @eugene_teoh @JafarUruc @SridharSola

4

7

34

Stephen James

@stepjamUK

5 years

#RLBench has been accepted to @icra2020 ! 🎉 We also have an active and growing community with >230 stars on GitHub! Keep the issues and pull requests coming! 🤖

Stephen James

@stepjamUK

5 years

We are thrilled to announce RLBench: an ambitious large-scale benchmark and learning environment for vision-guided manipulation with 100 unique, hand-design tasks! Paper: Video: w/ @StephenLJames , Z. Ma, D. Arrojo, @AjdDavison

2

40

114

0

3

33

Stephen James

@stepjamUK

6 years

Papers like these are incredibly valuable to the #robotics community, at a time when end-to-end approaches are becoming more prominent. This paper compares ORB-SLAM to a learned system (what I like to call 'implicit #SLAM '). Paper by @ducha_aiki :

1

10

30

Stephen James

@stepjamUK

5 years

We are happy to announce that the code for TecNets is now publicly available. Code: Paper:

Task-Embedded Control Networks for Few-Shot Imitation Learning

Much like humans, robots should have the ability to leverage knowledge from previously learned tasks in order to learn new tasks quickly in new and unfamiliar environments. Despite this, most...

arxiv.org

Stephen James

@stepjamUK

6 years

In our oral at #CoRL2018 tomorrow, I will present Task-Embedded Control Networks, which employ ideas from metric learning in order to create a task embedding that can be used by a robot to learn new tasks from one or more demonstrations. #robotics #MachineLearning #AI #phdlife

2

12

52

0

2

33

Stephen James

@stepjamUK

2 years

New! Auto-Lambda is a gradient-based meta learning framework that explores continuous, dynamic task relationships via task-specific weightings; achieving SOTA on CV and robotics tasks! Honoured to be one of the first set of accepted papers at @TmlrOrg ! Lead by @liu_shikun

Accepted papers at TMLR

@TmlrPub

2 years

Auto-Lambda: Disentangling Dynamic Task Relationships Shikun Liu, Stephen James, Andrew Davison, Edward Johns

0

4

30

1

7

32

Stephen James

@stepjamUK

2 years

Finally, after > year of hard work, we are excited to release the most ambitious Sim2Real project to date: End-to-end off-road autonomous driving, trained in sim, and transferred to the real world. @corl_conf Amazing work lead by @amberxie_ & @johnrso_ .

Amber Xie

@amberxie_

2 years

Excited to release our CoRL 2022 work, Sim-to-Seg: end-to-end RL w/ sim-to-real transfer through learned image segmentations! 🚗⛰ Work done with @johnrso_ , @NASAJPL , @pabbeel @stepjamUK 📜: 🌐:

1

17

88

2

8

32

Stephen James

@stepjamUK

11 months

One thing I've always found hacky with robot manipulation is the need to "disable" collision checking when interacting with objects (e.g. grasping, pushing, etc); now you don't have to! We propose a new domain: Language-Conditioned Path Planning! #CoRL2023

Amber Xie

@amberxie_

11 months

Excited to share our CoRL 2023 paper: Language-Conditioned Path Planning! ⚡️ Not all collisions are created equal! We address the limitations of traditional path planning by incorporating contact awareness into path planning. w/ @YoungwoonLee @pabbeel @stepjamUK

1

14

83

1

32

Stephen James

@stepjamUK

2 years

Our work on Sim2Real 6D object pose estimation via iterative self-training has been accepted to #ECCV2022 ! Congratulations to lead author Kai Chen!

Stephen James

@stepjamUK

2 years

Our new Sim2Real 6D Object Pose Estimation work! 1) Train pose estimation model on sim data. 2) Use model to generate poses on unlabelled real data. 3) Auto filter generated poses and update model. 4) Repeat 2-4. Result: SOTA performance + robot demo! 🤖

4

37

178

0

5

31

Stephen James

@stepjamUK

5 years

Having robot learning blues after the end of #CoRL2019 ? Fear not! Our new paper takes sim2real further and explores how we can learn to one-shot imitate humans (using TecNets) without any real-world data during training!

1

10

30

Stephen James

@stepjamUK

2 years

Kicking off @CVPR with coffee and a long zig-zag registration line. DMs are open if you want to meet up for a chat!

2

0

30

Stephen James

@stepjamUK

2 years

New work! 1) Pre-train action-free video prediction from "wild" data (e.g. RLBench/youtube/etc). 2) Fine-tune action-conditioned video prediction to quickly learn vision-based RL in a new domain (e.g. MetaWorld). Lead: @younggyoseo w/ @kimin_le2 , @pabbeel

Reinforcement Learning with Action-Free Pre-Training from Videos

Recent unsupervised pre-training methods have shown to be effective on language and vision domains by learning useful representations for multiple downstream tasks. In this paper, we investigate...

arxiv.org

Younggyo Seo

@younggyoseo

2 years

Can we leverage diverse out-of-domain videos for improving vision-based RL? Yes! Excited to share APV that quickly learns world models 🌎 by fine-tuning a pre-trained action-free video prediction model. Paper: w/ @kimin_le2 @stepjamUK @pabbeel

3

21

94

1

7

28

Stephen James

@stepjamUK

3 months

In 5 mins I will be talking about Next-best Pose Agents at the ICRA Workshop on 3D Visual Representations for Robot Manipulation. I'll be covering a few pieces of work, including our recent CVPR work: Hierarchical Diffusion Policy 🤖 #ICRA2024

Stephen James

@stepjamUK

5 months

Hierarchical diffusion policy is another step along the journey of making hierarchical next-best pose agents more capable, through introduction of a kinematically-aware low-level diffusion planner.🤖 New work from the Dyson Robot Learning Lab. CVPR 2024

1

22

137

0

3

26

Stephen James

@stepjamUK

2 years

Great talk by @sainingxie at #CVPR2022 , reminding us that if you add in all the training bells & whistles that vision transformers get, then ConvNets can perform equally as well / better.

0

2

27

Stephen James

@stepjamUK

1 year

#ICRA2023 may be over, but that doesn't mean we have to stop talking about #robots , right? My first time in York today to talk about "Hyper-efficient End-to-end Manipulation" at the @UniOfYork Robotic Manipulation Seminar. 🤖

1

5

25

Stephen James

@stepjamUK

2 years

Best paper award @CVPR ! Congrats to authors 👏

0

23

Stephen James

@stepjamUK

5 years

#RLBench tasks/features (3/n). This bin emptying task is one of the 100 tasks of RLBench. The goal is to move the objects from the grey bin to one of the specified coloured bins. The robot must infer the goal from either the textual description of the task or from the demos.

0

3

24

Stephen James

@stepjamUK

9 months

Great to be here in Atlanta for @corl_conf ! Join Amber Xie on Wednesday 5.15pm, where she will be presenting language-conditioned path planning --- a new way to do intelligent and intuitive path planning! #CoRL2023

Stephen James

@stepjamUK

11 months

One thing I've always found hacky with robot manipulation is the need to "disable" collision checking when interacting with objects (e.g. grasping, pushing, etc); now you don't have to! We propose a new domain: Language-Conditioned Path Planning! #CoRL2023

1

32

0

2

22

Stephen James

@stepjamUK

5 years

Robot infers action models by observing teacher demonstrations which are subsequently used for Monte Carlo tree search. Tasks include: placing box inside box and taking cereal out of cabinet. Paper: Tim Welschehold, ..., @wolfram_burgard

0

2

23

Stephen James

@stepjamUK

3 years

We present End-to-End Egospheric Spatial Memory (ESM) at #ICLR2021 , today 9-11am PST! Working with a wrist camera on a manipulator? ESM represents the sequence data as an ego-sphere 'memory' around the camera, outperforming LSTM and NTM on IL and RL tasks!

1

23

Stephen James

@stepjamUK

3 years

We are releasing Ivy! A rich set of functions (compatible with TensorFlow, PyTorch, Jax, MXNet, and Numpy) that complements your research in vision, robotics, and more! Ivy can significantly reduce LoC for rapid prototyping. Code below is how I like to use Ivy!

Unify

@letsunifyai

3 years

We are excited to release Ivy, a new open source Deep Learning framework! Ivy unifies the syntax and call signatures of existing frameworks. Write your code once in Ivy, and support all frameworks simultaneously. Links to Paper, Code and Docs here:

26

75

334

1

3

22

Stephen James

@stepjamUK

5 years

#RLBench tasks/features (2/n). RLBench comes with support for domain randomisation of both visual and dynamics with 1 line of code, allowing for rapid sim-to-real experiments!

0

4

22

Stephen James

@stepjamUK

22 days

If you are interested in sample-efficient imitation learning, come and watch Vitalis present his work: Render and Diffuse (R&D)! 8.30am #RSS2024

Vitalis Vosylius

@vitalisvos19

22 days

Today we'll be presenting R&D at #RSS2024 during the Imitation Learning session at 8:30 am -- come by and visit our poster afterwards if you are around! 🤖

0

4

19

0

22

Stephen James

@stepjamUK

4 years

After many requests, #RLBench task building video tutorials are here! This complements the text-based tutorials that are already available on GitHub.

RLBench Task Building Tutorials

www.youtube.com

0

4

20

Stephen James

@stepjamUK

2 years

We have released TECO! A video prediction model that excels in generating long, temporally consistent videos in complex 3D scenes! Work led by @WilsonYan8 📜 🌐

Temporally Consistent Transformers for Video Generation

To generate accurate videos, algorithms have to understand the spatial and temporal dependencies in the world. Current algorithms enable accurate predictions over short horizons but tend to suffer...

arxiv.org

Wilson Yan

@wilson1yan

2 years

Excited to announce TECO, an efficient video prediction model that can generate long, temporally consistent video for complex datasets in 3D scenes such as DMLab, Minecraft, Habitat, and real-world video from Kinetics! 📜 🌐 (🧵)

8

61

406

0

2

19

Stephen James

@stepjamUK

2 years

Workshop on Pre-training Robot Learning final call for papers! The second and final submission window closes today (Oct 26th) at 11:59PM UTC! Submit your 4-page extended abstract; virtual-only presentations also accepted!

Stephen James

@stepjamUK

2 years

Announcing the 1st "Workshop on Pre-training Robot Learning" at @corl_conf , Dec 15. Fantastic lineup of speakers: Jitendra Malik, Chelsea Finn, Joseph Lim, Kristen Graumen, Abhinav Gupta, Raia Hadsell. Submit your 4-page extended abstract by September 28.

3

33

116

0

6

18

Stephen James

@stepjamUK

2 years

This study took ~year to complete. We had no horse in this race, and tried to make it as fair as possible. We hope it's useful to the community, and hope that it encourages future meta-RL papers to include a simple multi-task baseline; you may be surprised how well it works! 😉

0

17

Stephen James

@stepjamUK

4 years

We have released the code to our #CVPR2020 paper, MoreFusion!

Kentaro Wada

@wkentaro_

4 years

Released the code of our paper, MoreFusion #CVPR2020 ! It contains all of the software for training/evaluation of pose estimation network, joint pose refinement, and online demonstration with an RGB-D camera/robot.

2

26

110

0

4

15

Stephen James

@stepjamUK

27 days

The successor to RLBench?? This is a (very hard) robot learning benchmark for mobile (biped) bi-manual manipulation. Like RLBench, it comes with vision, demos and sparse rewards!

Nikita Chernyadev

@nc__dev

27 days

🚀 Looking for a benchmark for bi-manual mobile manipulation with nicely collected demonstrations? We are excited to release BiGym, a new benchmark with human-collected demos! 🌐 Website: 📄 Paper: 💻 Code:

3

17

39

1

16

Stephen James

@stepjamUK

27 days

A new RL framework: Coarse-to-Fine Reinforcement Learning, along with a new algorithm within this framework: Coarse-to-fine Q-Network (CQN).

Younggyo Seo

@younggyoseo

27 days

Introducing CQN: Coarse-to-fine Q-Network, a value-based RL algorithm for continuous control🦾Initialized with 20~50 demonstrations, it learns to solve real-world robotic tasks within 10 mins of training, without any pre-training and shaped rewards! (1/4)

1

32

109

1

16

Stephen James

@stepjamUK

2 years

The gist of our paper is to show *how* we can leverage the power of Bingham for RL! 💪 When evaluating our approach on the Wahba problem and a set of vision-based robot manipulation tasks from RLBench, we achieve superior performance over a Gaussian parameterization! 🧵5/5

0

15

Stephen James

@stepjamUK

27 days

Generative Image as Action Models. Represent robot actions by “drawing” joint actions into the image.

Mohit Shridhar

@mohito1905

27 days

Image-generation diffusion models can draw arbitrary visual-patterns. What if we finetune Stable Diffusion to 🖌️ draw joint actions 🦾 on RGB observations? Introducing 𝗚𝗘𝗡𝗜𝗠𝗔 paper, videos, code, ckpts: 🧵Thread⬇️

2

32

152

1

16

Stephen James

@stepjamUK

6 years

See, feel, act: A robot learns to play #Jenga via a hierarchical learning approach. Paper: Video: Work by @MIT . #Robotics #AI

0

3

13

Stephen James

@stepjamUK

27 days

Collect all your robot data with a green screen, and then apply Chroma Keying. Leads to policies that can generalize to ANY visually distinct location (scene)! The first example of a truly (scene) general robot learning policy?

Eugene Teoh

@eugene_teoh

27 days

🚀 We are excited to announce GreenAug (Green-screen Augmentation), a physical visual augmentation method for robot learning algorithms! GreenAug enables generalisation to unseen visually distinct locations (scenes). In collaboration with @TinkerSumit @yusufma555 @stepjamUK (1/6)

1

14

34

1

2

14

Stephen James

@stepjamUK

5 years

@AjdDavison RLBench can facilitate research in: few-shot learning, reinforcement learning, imitation learning, multi-task learning, geometric computer vision, etc. Observations: rgb, depth, and segmentation masks from an over-the-shoulder stereo camera and an eye-in-hand monocular camera.

0

1

13

Stephen James

@stepjamUK

2 years

Quaternions are often used as the output rotation representation when using deep networks, but due to their antipodal symmetric property, sampling a quaternion from a Gaussian doesn't seem appropriate. Here comes the Bingham distribution to the rescue! 🧵2/5

1

13

Stephen James

@stepjamUK

6 years

Uber has announced Atari Zoo () which is collection of trained #reinforcementlearning agents, along with this awesome visualisation tool: . Super cool, worth checking out!

Creating a Zoo of Atari-Playing Agents to Catalyze the Understanding of Deep Reinforcement Learning...

Uber AI Labs releases Atari Model Zoo, an open source repository of both trained Atari Learning Environment agents and tools to better understand them.

www.uber.com

0

4

12

Stephen James

@stepjamUK

2 years

This is the first work (I'm aware of) that does sim2real via iterative self-training! Great collaboration between UC Berkeley and the Chinese University of Hong Kong. Lead: Kai Chen

3

11

Stephen James

@stepjamUK

5 years

Well needed rest leading up to #CoRL2019 . Visited Tokyo, Kyoto, and now Osaka. Looking forward to 3 days of exciting work! 🤖

0

11

Stephen James

@stepjamUK

2 years

My final PhD project from the Dyson Robotics Lab at Imperial College London. Work with @wkentaro_ , Tristan Laidlow, and @AjdDavison . 2 new papers extending Q-attention coming out very soon... 😉

2

0

11

Stephen James

@stepjamUK

6 years

Jan Matas presenting work at #CoRL2018 in collaboration with myself and @AjdDavison , where we learn to transfer deformable object manipulation policies from simulation to the real world. Paper: #ReinforcementLearning #manipulation #simtoreal

0

11

Stephen James

@stepjamUK

2 years

New work! Dealing with fragile objects? Many grasping systems extract objects *without* considering potential damage to others (left vid). Our method combines object-level mapping & safety-aware Q-learning motion planning to safely extract occluded objects from piles (right vid).

Kentaro Wada

@wkentaro_

2 years

Released our work on object extraction: SafePicking #ICRA2022 ! This robotic system finds and “safely” extracts target objects via known object mapping with an onboard camera. Check out the project page for paper/code: Work with @stepjamUK @AjdDavison (1/n)

1

42

124

0

1

11

Stephen James

@stepjamUK

6 years

Quadrupedal robot ( #ANYmal ) learns to walk and recovering from falling in complex configurations in simulation, with zero-shot transfer to the real world! I went to Jemin's #PhD defence, and was very impressed by the physics simulation he had created.

1

10

Stephen James

@stepjamUK

6 years

Finally taken the plunge to Twitter, as I seem to be the only PhD student who does not use it. Will try to be fairly active, but no promises. #PhD #Robotics #MachineLearning

1

11

Stephen James

@stepjamUK

1 year

What an incredible start to #ICRA2023 ! Catch me at: 📅 Today 10.20am: Introducing Dyson Robot Learning Lab at Exhibition Hall. 📅 Friday 4pm: Presenting pretraining + sample-efficient robot learning at L-DOD workshop. 📅 Monday 9.10am: Talking at York Robot Manipulation Seminar.

0

1

11

Stephen James

@stepjamUK

3 years

Ivy was used to implement our recently accepted ICLR 2021 paper: End-to-End Egospheric Spatial Memory (ESM). ESM encodes the scene into an egosphere, which travels with an agent. Great work led by @DanielLenton1 Site: Video:

End-to-End Egospheric Spatial Memory

Published at the International Conference on Learning Representations (ICLR) 2021End-to-End Egospheric Spatial Memory (ESM) encodes the memory in an ego-sphe...

www.youtube.com

0

1

11

Stephen James

@stepjamUK

2 years

Robots at #CVPR2022 ; a rare sight! 🙂

Pete Florence

@peteflorence

2 years

Happening tomorrow — join us online or in New Orleans!

0

3

43

0

11

Stephen James

@stepjamUK

2 years

Another surprising result: we can solve novel vision-based, sparse-reward manipulation (test-time) tasks *without* any demonstrations, by pretraining on only 10 tasks! This is a big win for robotics -- we may need demonstrations for pre-training, but not for fine-tuning!

1

0

11

Stephen James

@stepjamUK

4 years

I have quite a few qualms with MuJoCo, and this is certainly one of the main ones. Might be a good time for folks to check out #RLBench for benchmarking their manipulation algorithms. 😉 () Uses Bullet under the hood.

RLBench

RLBench: The Robot Learning Benchmark & Learning Environment Stephen James Zicong Ma David Rovick Arrojo Andrew J. Davison Dyson Robotics Lab, Imperial College London IEEE Robotics and Automation...

sites.google.com

Erwin Coumans 🇺🇦

@erwincoumans

4 years

Glad to see some discussion about the rejection of the use of proprietary tech (MuJoCo, Unity etc) in public benchmarks, while open source alternatives such as PyBullet are available:

10

19

144

0

10

Stephen James

@stepjamUK

6 years

Indexing Datasets of 3D Indoor Objects: A blog post that discusses the strengths and limitations of various 3D datasets that have been released over the past 15 years.

sim2real

@sim2realAIorg

6 years

Part I: Indexing Datasets of 3D Indoor Objects Blog:

1

35

84

0

2

10

Stephen James

@stepjamUK

2 years

@_akhaliq @_akhaliq More info on Sim2Seg here:

Stephen James

@stepjamUK

2 years

Finally, after > year of hard work, we are excited to release the most ambitious Sim2Real project to date: End-to-end off-road autonomous driving, trained in sim, and transferred to the real world. @corl_conf Amazing work lead by @amberxie_ & @johnrso_ .

2

8

32

0

1

9

Stephen James

@stepjamUK

2 years

Occam's razor at work

Josh Tobin

@josh_tobin_

2 years

Sometimes in ML simple things just work. Pre-training + fine-tuning is one of those hard-to-beat baselines

0

1

32

0

1

10

Stephen James

@stepjamUK

2 years

... and six different tasks on a 7-DoF robot arm.

1

9

Stephen James

@stepjamUK

6 years

A suprise visit from Satya Nadella ( #Microsoft CEO) at #CoRL2018 . Talking about #AI , #Security , and #Privacy issues that arise in emerging technologies.

0

9