Thrilled to announce Octo 🐙, an open-source robot foundation model! Octo is a sota generalist robot policy based on transformer+diffusion. Most importantly, you can finetune Octo *today* with flexible observation and action spaces on your robot setup!
Excited to start my Postdoc with Prof.
@svlevine
at
@UCBerkeley
! I look forward to collaborating with all the amazing researchers at
@berkeley_ai
(BAIR)!
Today I had the pleasure of presenting my 🤖 robot learning research at
@ToyotaResearch
in Los Altos and to learn about their ongoing efforts on building and scaling 📈 AI models for robotics! Thanks for the invitation!
Excited that AVLMaps, our take for 🤖 following multimodal prompts from audio 🔊, vision 👀 and language🗣️ w/ foundation models to solve zero-shot spatial goal Navigation will be presented at ISER 23!
w/
@huang_chenguang
@andyzeng_
@wolfram_burgard
:
Honored to receive the IEEE Robotics and Automation Letters Best Paper Award for our CALVIN paper and being a runner up for the IEEE International Conference on Robotics and Automation Best Paper Award in Robot Learning for HULC2!
w/
@wolfram_burgard
@UniFreiburg
@utn_nuremberg
Join us for the 2nd edition of the
#LangRob
workshop at
#CoRL2023
in vibrant Atlanta! Get ready for an unforgettable day with an all-star ensemble of speakers and two spicy panels that will ignite your passion for language and robotics! 🔥🤖
P.S. Guess who wrote this tweet 😉
It was an honor to give a Google Tech Talk to present my latest research on 🤖 robot learning at
@GoogleDeepMind
in Mountain View! It was fun and very inspirational to spend the day talking to many smart people about tackling the next research challenges for scaling 🤖 learning!
Join us for the 2nd edition of the
#LangRob
workshop at
#CoRL2023
in vibrant Atlanta! Get ready for an unforgettable day with an all-star ensemble of speakers and two spicy panels that will ignite your passion for language and robotics! 🔥🤖
P.S. Guess who wrote this tweet 😉
Excited to share our latest collaboration with
@GoogleAI
on following multimodal prompts from audio, vision and language to solve zero-shot spatial navigation tasks in the real world!
Can robots 🤖 to navigate to sounds 🔊 they've heard?
w/ audio-language 🔊✏️ foundation models, excited that we can now ask our helper robots to "go to where you heard coughing"
Audio-Visual-Language Maps w/
@huang_chenguang
@oier_mees
@wolfram_burgard
:
PhD thesis submitted! Under the supervision of
@wolfram_burgard
Big shout out to my family, friends, and colleagues for the incredible encouragement and support. This milestone wouldn't have been possible without you! 🙌🎓
#PhD
#ThesisSubmitted
Unlabeled ♾ data is the key ingredient of todays ML, but how can we leverage unstructured, unlabeled, offline datasets in robot learning?
Introducing TACO-RL, a robotic model that consumes uncurated data to learn multi-tier, long-horizon policies!
📜
Congrats to my student
@ErickRoseteBeas
who has won the Best Master Thesis Award from the Association of German Engineers (VDI) ! 🙌 His thesis "Skill-Chaining Latent Behaviors with Offline RL" co-supervised with
@GabrielKalweit
led to our TACO-RL
#CoRL2022
paper!
Unlabeled ♾ data is the key ingredient of todays ML, but how can we leverage unstructured, unlabeled, offline datasets in robot learning?
Introducing TACO-RL, a robotic model that consumes uncurated data to learn multi-tier, long-horizon policies!
📜
Honored to have been invited to serve as an Associate Editor of Robot Learning for
@ieeeras
Robotics and Automation Letters journal (RA-L)! Looking forward to working with the team of
@AleksandraFaust
🙂
Habemus PhD! 🎓 Thanks to
@wolfram_burgard
, Dieter Fox, Joschka Boedecker, Matthias Teschner and Armin Biere for being on my committee. Thanks to all friends, family and collaborators for being part of this incredible journey. Summa cum laude!
I am thrilled to receive the award of AI Newcomer 2023 by the German Federal Ministry of Education and Research and the German Informatics Society!
Thanks to my collaborators, all the people who supported us with their votes, the jury and the KI Amp organizers
@informatikradar
Oier Mees is AI Newcomer in the category Technical and Engineering Sciences - congratulations! 🎉
@oier_mees
research aims to enable robots to perform a wide range of everyday tasks in human-centered environments. More information here:
Join us for the 2nd edition of the
#LangRob
workshop at
#CoRL2023
in vibrant Atlanta! Get ready for an unforgettable day with an all-star ensemble of speakers and two spicy panels that will ignite your passion for language and robotics! 🔥🤖
P.S. Guess who wrote this tweet 😉
Excited to have been part of this 🌎 scale collaboration! Ground breaking 173 researchers, 34 labs, 22 embodiments and +1M trajectories. Hope this will become our ImageNet moment!
Proud to have been the 1. 🇪🇺 lab to provide evals in my Freiburg setup before joining Berkeley.
If you want to know how to use the Open X-Embodiment AKA RT-X dataset check out the deep dive by the fantastic
@KarlPertsch
who led the herculean effort of collecting and standardizing the individual datasets together with the no less amazing
@QuanVng
and
@xiao_ted
!
Very excited to release the Open X-Embodiment Dataset today — the largest robot dataset to date with 1M+ trajectories! Robotics needs more data & this is a big step!
There’s lots to unpack here, so let’s do a deep dive into the dataset!
🧵1/15
On my way to
#IROS2022
in Kyoto 🇯🇵 to present to present our RA-L papers on language conditioned policy learning CALVIN and HULC! 👇
Ping me if you are interested in chatting on robot skill learning and/or language grounding!
VLMaps is a map representation that fuses pre-trained visual-language embeddings into a 3D reconstruction of an environment, enabling robots to index landmarks and generate open-vocabulary maps for path planning. Learn more and copy the code →
We are hosting the 2nd workshop on Language and Robot Learning at
#corl2023
! We've got an incredible panel of speakers, and we are excited to discuss together "Language as Grounding"! Be sure to submit your papers!
Announcing the 2nd Workshop on Language and Robot Learning at
#CoRL2023
on November 6th, 2023! This year's theme is "Language as Grounding".
Featuring a great speaker lineup and two panels!
Website:
CfP:
Deadline: October 1
So you say that LLMs can't be used for lower-level robot control, because they aren't directly trained on robotics data?
Actually, we found that often, they can!
"Language Models as Zero-Shot Trajectory Generators".
See: .
I'll explain more below ... 🧵
Join us for the 2nd edition of the
#LangRob
workshop at
#CoRL2023
in vibrant Atlanta! Get ready for an unforgettable day with an all-star ensemble of speakers and two spicy panels that will ignite your passion for language and robotics! 🔥🤖
P.S. Guess who wrote this tweet 😉
Excited and deeply grateful to share that I've been selected as a recipient of the 2023 Google PhD Fellowship! Thank you
@GoogleAI
for graciously supporting my research in developing intelligent, self-improving real-world systems 🤖
What if the form of visuomotor policy has been the bottleneck for robotic manipulation all along? Diffusion Policy achieves 46.9% improvement vs prior StoA on 11 tasks from 4 benchmarks + 4 real world tasks! (1/7)
website :
paper:
Got a chance to dig through the big robot X-embodiment dataset released last week, and hacked together a little website for others to look through the data.
Check it out! There's some pretty random and diverse robot data in there
Join us for the 2nd edition of the
#LangRob
workshop at
#CoRL2023
in vibrant Atlanta! Get ready for an unforgettable day with an all-star ensemble of speakers and two spicy panels that will ignite your passion for language and robotics! 🔥🤖
P.S. Guess who wrote this tweet 😉
From our demo floor at AI@, check out Code as Policies at work. This helper robot is able to compute and execute a task given via natural language. Read more →
Instead of finding the perfect prompt for an LLM (let's think step by step), you can ask LLMs to critique their outputs and immediately fix their own mistakes. Here's a fun example:
Applications for the
#RSSPioneers2023
workshop (to be held at
#RSS2023
) are open! Senior PhD students and early-career researchers in robotics are encouraged to apply. Find more details on how to apply here:
This summarizes GPT-4s 98 pages of technical report perfectly: we trained stuff and got results. It's sad that AI is increseangly becoming closed source research.
We learn a single multi-task visuomotor policy for 25 tasks that outperforms state-of-the-art baselines, with an order-of-magnitude improvement by taking a 9 hour long continuous demo and using only image obs.
We have released our data, code and models!
@xiao_ted
You are too kind
@xiao_ted
! All this work stands on the shoulders of many of your and your teams (
@coreylynch
, Pierre Sermanet...) advancements!
A key ingredient for our offline RL policy is the goal sampling strategy. We find that incorporating hard negative examples from scenes that contains similar proprioceptive states enables the Q-function to learn to focus on the scene’s under-actuated parts (e.g., objects).
TACO-RL makes sense of this data by combining the strengths of imitation learning and offline RL. Our hierarchical approach combines a low-level policy that learns latent skills via imitation and a high-level policy learned from offline RL for skill-chaining the latent behaviors.
Due to popular demand, we are extending the
#CoRL2023
LangRob workshop deadline by one week to *October 8, 23:59 AOE*!
LangRob remains non-archival, and welcomes recent submissions to ICRA/ICLR.
We are hosting the 2nd workshop on Language and Robot Learning at
#corl2023
! We've got an incredible panel of speakers, and we are excited to discuss together "Language as Grounding"! Be sure to submit your papers!
So far, there have been some remarkable large-scale robotic learning results, datasets, and milestones this year. But we have something pretty big coming out tomorrow. So big that we needed a globe to visualize its scale😉
Had a blast presenting TACO-RL, our take on scaling robot learning to highly diverse, unstructured, unlabeled, offline data to learn task-agnostic, long-horizon policies at
#CoRL2022
!
@ChukwuemekaOme4
@LerrelPinto
That's a great question! A good starting point might be CALVIN, which contains several hours of multimodal play data for 4 simulated tabletop environments.
@hausman_k
Amazing work
@hausman_k
@xiao_ted
et. al. ! Since you mention the project took so long, what were the biggest challenges you had to solve to get it work?
Based on the success of tapping into large scale unlabeled data in other fields, our goal was to build a model that absorbs uncurated robot data. It should discover skills from highly diverse, multimodal data, collected by asking users to teleop without any tasks in mind.