Very excited to release OpenVLA today, a 7B parameter open-source vision-language-action model (VLA).
🦾 SoTA generalist policy (better than Octo & RT-2-X)
⚡️ Easy to run & fine-tune on 1 GPU with quantization and LoRA
💻 Open-source PyTorch codebase
🤗 Models on HuggingFace
1/
Very excited to release the Open X-Embodiment Dataset today — the largest robot dataset to date with 1M+ trajectories! Robotics needs more data & this is a big step!
There’s lots to unpack here, so let’s do a deep dive into the dataset!
🧵1/15
3 mo. ago we released the Open X-Embodiment dataset, today we’re doing the next step:
Introducing Octo 🐙, a generalist robot policy, trained on 800k robot trajectories, stronger than RT-1X, flexible observation + action spaces, fully open source!
💻:
/🧵
Access to *diverse* training data is a major bottleneck in robot learning. We're releasing DROID, a large-scale in-the-wild manipulation dataset. 76k trajectories, 500+ scenes, multi-view stereo, language annotations etc
Check it out & download today!
💻:
Our OpenVLA model has been downloaded more than 20k times in less than a month -- the most for any robotics model on the 🤗 hub by a long shot!
Here is a little "cook book" for people who want to get started using OpenVLA! 🧑🍳
1/🧵
Our OpenX paper won best paper at ICRA! Congrats to all my co-authors! 🎉🎉
This is an ongoing effort, we recently added new datasets from the community that double the size of the OpenX dataset -- keep 'em coming! :)
Check datasets & how to contribute:
Octo has been accepted to RSS and we finally arxiv'd the paper! 🐙
Many small updates vs the December release: more ablations, new checkpoints, code fixes etc
👇
Octo
An Open-Source Generalist Robot Policy
Large policies pretrained on diverse robot datasets have the potential to transform robotic learning: instead of training new policies from scratch, such generalist robot policies may be finetuned with only a
It was fun to present Open X-Embodiment & RT-X at CoRL today with
@QuanVng
! We were very excited about the initial release of the Open X-Embodiment dataset, but it's just the start! We covered lots of open problems in the talk as well👇
Excited to present STAR, our work on cross-domain imitation
@corl_conf
!
Our goal: use demonstrations across domains, e.g. from robot in kitchen A to robot in kitchen B, or even from human to robot. With STAR I can teach a robot new tasks with videos recorded in my kitchen!
🧵👇
It's awesome to see the positive community response to our release! We're getting inquiries from around the world to contribute more data -- wheeled robots, drones, humanoids etc! 🚀🚀🚀
Please keep them coming 🙂
open-x-embodiment
@googlegroups
.com
Very excited to release the Open X-Embodiment Dataset today — the largest robot dataset to date with 1M+ trajectories! Robotics needs more data & this is a big step!
There’s lots to unpack here, so let’s do a deep dive into the dataset!
🧵1/15
Evaluation of robot foundation models is a huge challenge: imagine running robot rollouts across 100s of scenes + tasks + embodiments. How can we make eval keep up w/ model improvements?
Introducing SIMPLER: sim eval envs for your favorite real robot foundation models!
Short 🧵
Scalable, reproducible, and reliable robotic evaluation remains an open challenge, especially in the age of generalist robot foundation models. Can *simulation* effectively predict *real-world* robot policy performance & behavior?
Presenting SIMPLER!👇
Excited to release our work on Embodied Chain-of-Thought Reasoning today!
We can boost performance of vision-language-action models like OpenVLA by a large margin without any additional robot training data!
The key: simply think before you act!
1/
🤖Can robots think through complex tasks step-by-step like language models?
We present Embodied Chain-of-Thought Reasoning (ECoT): enabling robots to reason about plans and actions for better performance🎯, interpretability🧐, and generalization🌎.
See .
Robot learning needs data, but collecting it is expensive. How can we make the most of existing datasets?
In SPRINT, we use LLMs to auto-augment language instructions on robot datasets. Our agents learn a lot more tasks during pre-training *for free*!
See Jesse’s 🧵for details!👇
Having humans annotate data to pre-train robots is expensive and time-consuming!
Introducing SPRINT:
A pre-training approach using LLMs and offline RL to equip robots w/ many language-annotated skills while minimizing human annotation effort!
URL:
🧵👇
New paper on *Skill-based Learning with Demonstrations (SkiLD)*!
While current imitation learning follows the _low-level actions_ in the demos, SkiLD follows the demonstrated _skills_.
SkiLD enables efficient demo-guided RL & imitation learning on long-horizon tasks!
1/N
Shoutout to the folks at Rerun who built a visualizer for our DROID dataset -- looks very cool! Allows you to visualize the point cloud from our multi-view stereo cams as well! And should work for any new dataset collected on the DROID robot platform!
Thanks
@rerundotio
:)
A Rerun Viewer for the DROID Dataset!
DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset is a robot manipulation dataset by
@SashaKhazatsky
et al. with 76k demonstration trajectories or 350h of interaction data, collected across 564 scenes and 86 tasks.
New work on scaling robot learning from the team I work with at Google!
Especially excited about RT1’s capability to ingest data from diverse sources, eg sim or even experience from other robots + demonstrate transfer -- very useful for scaling robotic dataset size & diversity!
Introducing RT-1, a robotic model that can execute over 700 instructions in the real world at 97% success rate!
Generalizes to new tasks✅
Robust to new environments and objects✅
Fast inference for real time control✅
Can absorb multi-robot data✅
Powers SayCan✅
🧵👇
Grateful to be awarded the best paper presentation award
@corl_conf
! 🎉
Huge credit goes to all my lab mates @ CLVR lab, particularly to my co-author
@YoungwoonLee
, for all the tireless feedback that greatly improved the talk! :)
Talk recording:
Data collection is a major bottleneck in robot learning: it’s mostly done w/ tedious & expensive human teleoperation. Can we use learning to make data collection itself more efficient?
Introducing PATO, our approach for scalable robot data collection w/ learned assistive policies
Excited to present PATO: Policy Assisted TeleOperation, our recent work on scaling robot data collection!
PATO uses a policy trained on prior data to assist the user during data collection, making teleop easier and even allows to teleop multiple robots simultaneously.
🧵👇
How can we use large offline datasets for accelerating the learning of new tasks? We can transfer skills!
Check out our
#CoRL2020
paper on efficient skill transfer with learned skill priors!
📄Paper:
💻Website & Code:
Thread👇(1/8)
This looks awesome! Simulation can be a valuable tool for robot data scaling & eval, but the hard part is building diverse simulation envs AND datasets. Glad to see Soroush et al's sim data line of work expanded to more diverse envs! Excited to give this a try!
I’m excited to introduce RoboCasa, a large-scale simulation framework for everyday tasks. Scaling is the key driving force to unlocking generalist robots, and RoboCasa leverages simulation to take scaling to a whole new level. A short 🧵
If you want to browse through the Open X-Embodiment data, but don't like fiddling with Colabs, check out this neat website
@its_dibya
built that gives you a quick overview of all datasets!
Got a chance to dig through the big robot X-embodiment dataset released last week, and hacked together a little website for others to look through the data.
Check it out! There's some pretty random and diverse robot data in there
Check out Lucy's new project! Finally, every roboticist's favorite pastime, "yelling at your robot", can be useful for once!
Bonus: lots of ALOHA trail mix in the lab! 😍
Introducing Yell At Your Robot (YAY Robot!) 🗣️- a fun collaboration b/w
@Stanford
and
@UCBerkeley
🤖
We enable robots to improve on-the-fly from language corrections: robots rapidly adapt in real-time and continuously improve from human verbal feedback.
YAY Robot enables
Glad to see RT-2 out! We show that VLM backbones are a great way to equip policies with robustness from internet-scale data.
RT-2 strongly improves the generalization ability of existing skills (eg new scenes / objects) -- learning new low-level behaviors is the next frontier!
PaLM-E or GPT-4 can speak in many languages and understand images. What if they could speak robot actions?
Introducing RT-2: our new model that uses a VLM (up to 55B params) backbone and fine-tunes it to directly output robot actions!
Big FOMO! -- but you guys will rock the presentation :)
If you're @ ICRA, check out Quan's presentation of our Open X-Embodiment project today, nominated for a best paper award 🎉
Room: CC-Main Hall
Time: 10:30-12:00
Check out
@Jesse_Y_Zhang
's CoRL oral on LLM-guided skill learning.
Simple recipe: start from a base set of skills —> use LLM to guide exploration towards meaningful skill chains —> expand the skill library w/ RL.
We show that this "skill bootstrapping" phase helps downstream RL!
How can our robots autonomously practice **new tasks** in **new environments**?
Introducing BOSS: A reinforcement learning (RL) framework that trains agents to solve new tasks in new environments with LLM guidance!
**CoRL 2023 Oral**
🧵👇
Introducing 𝐃𝐫𝐢𝐯𝐞𝐕𝐋𝐌, VLM meets Autonomous Driving.
We propose a dual system that drives a car autonomously in complex driving scenarios.
- Slow system: VLM
- Fast system: classical AD pipeline
Enjoy our onboard demo!
Project Page:
Excited to be presenting SPiRL as an oral talk at today's plenary session on RL
@corl_conf
! Join to learn about skill priors for accelerated RL on new tasks!
Oral: Wed (today), 8:15am PST
Interactive: Wed, 12:30pm PST
w/
@YoungwoonLee
&
@JosephLim_AI
How can we use large offline datasets for accelerating the learning of new tasks? We can transfer skills!
Check out our
#CoRL2020
paper on efficient skill transfer with learned skill priors!
📄Paper:
💻Website & Code:
Thread👇(1/8)
@chris_j_paxton
@_ericrosen
Indeed existing x-embodiment models like RT-X/Octo don't align action spaces or condition on action space definition/URDF -- that's a major reason why they don't usually work 0-shot on new robot setups: they don't know what action space to use -- we're hoping to fix that soon! :)
Super cool work from Cheng et al! Robot data collection in the wild without the pain of moving robots around!
Before we deploy robots at scale + in the wild, this can greatly increase diversity of robot data + help overcome activation energy for getting generalizable policies
Can we collect robot data without any robots?
Introducing Universal Manipulation Interface (UMI)
An open-source $400 system from
@Stanford
designed to democratize robot data collection
0 teleop -> autonomously wash dishes (precise), toss (dynamic), and fold clothes (bimanual)
Interested in large task-agnostic datasets in robotics? We show how to effectively combine them w/ demonstrations for sample efficient learning of new tasks!
Presenting
@corl_conf
poster session 4 (Wed 11.30-12.30 GMT)!
📜:
💻:
New paper on *Skill-based Learning with Demonstrations (SkiLD)*!
While current imitation learning follows the _low-level actions_ in the demos, SkiLD follows the demonstrated _skills_.
SkiLD enables efficient demo-guided RL & imitation learning on long-horizon tasks!
1/N
Check out our new work on visual planning and control! Our model uses a divide-and-conquer strategy to break long-horizon planning problems into easier sub-problems, allowing us to solve tasks that require planning over hundreds of time steps!
This should be a great tutorial by Lerrel,
@notmahi
and
@RussTedrake
for anyone wanting to catch up on modern techniques for imitation learning!
Lots of the practical tips should transfer to fine-tuning of large pre-trained models too!
(see zoom link in Lerrel's thread)
This
#RSS2024
on July 19, we are organizing a tutorial on supervised policy learning for real world robots!
Talks by
@notmahi
&
@RussTedrake
will cover the fundamentals of imitation, recent algorithms, walk-through code, and practical considerations.
Excited to present two papers w/ co-authors at ICLR this week!
1⃣ Task-Induced Representation Learning:
We investigate representation learning in visually complex environments.
Q: How can we learn to represent important info & ignore distractors?
A: Use prior task experience!
Check out Lucy's and
@YoungwoonLee
's cool work on combining learned skills and model-based RL! Enables more sample efficient learning than model-free skill-RL approaches like SPiRL!
+ first skill-based RL results on the new CALVIN benchmark!
Lucy's first paper -- well done! :)
Can robots be farsighted? We introduce SkiMo (Skill + Model-based RL), which allows more accurate and efficient long-horizon planning through temporal abstraction. SkiMo learns temporally-extended, sparse-reward tasks with 5x fewer samples!
🧵👇
2D trajectories for task specification are more grounded than language, but easier to provide than goal images, eg by crowd workers / VLMs. + easy to relabel in hindsight + transfer nicely from human video!
Very cool work
@Jiayuan_Gu
@xiao_ted
et al!
Instead of just telling robots “what to do”, can we also guide robots by telling them “how to do” tasks?
Unveiling RT-Trajectory, our new work which introduces trajectory conditioned robot policies. These coarse trajectory sketches help robots generalize to novel tasks! 🧵⬇️
Out of the box, Octo can control multiple robots, use 3rd person + wrist cameras, language instructions & goal images.
Key feature: Octo can be quickly finetuned to use new observation & action spaces! In <5 hours on a 24 GB VRAM GPU!
2/
By training on in-the-wild human videos, we can use demonstrations from *unseen* environments, e.g. 3 mins of video recorded in my kitchen substantially accelerates RL in a new robot env in our experiments.
To show that the data is useful for learning, we trained a series of large-scale policies (RT-1-X, RT-2-X) & found co-training with our data to improve performance substantially! We’re releasing model checkpoints too, check Quan’s tweets for details!
11/
We will present our work on keyframe-based video prediction in the workshop on Task-agnostic RL (TARL) tomorrow afternoon. If you're at ICLR, come see us at our poster! (joint work with
@_oleh
,
@yiy0602
,
@CSProfKGD
, Joseph Lim,
@KostasPenn
,
@drew_jaegle
)
Creating this dataset was a huge community effort (look at that author list 😀)! I led the dataset construction and had calls with countless labs & everybody was very excited to contribute data — there is a lot of momentum in the community towards sharing & reusing data 🙂
12/
@_oleh
and I are presenting our work on hierarchical models for long-horizon prediction and planning at the
#BIGICML
workshop today, start is at 10.40PT. Come join us to chat about predictive models and model-based RL!
Here are the dataset resource links:
✅Colab (vis / download / data loaders):
✅Overview Sheet (filtering):
All data is fully open-source under a commercially usable CC-BY 4.0 license!
10/
Great work! 💯 lots of room to improve on the vision side of VLMs — robotics could be a great test bed too!
For VLA training (VLM+action) we found existing vision encoders need lots of fine-tuning to work well for robot control, though admittedly 🤖 eval isn’t straightforward 🥲
Introducing Cambrian-1, a fully open project from our group at NYU. The world doesn't need another MLLM to rival GPT-4V. Cambrian is unique as a vision-centric exploration & here's why I think it's time to shift focus from scaling LLMs to enhancing visual representations.🧵[1/n]
We assembled the dataset by pooling *existing* robot datasets from our collaborators @ Google and many many academic labs (34!).
In total we included 60 individual datasets with 22 different robot embodiments — many robot arms, bi-manual robots, quadrupeds, wheeled robots etc.
2/
The full dataset download is ~4.5 TB. We also provide a sheet that allows you to filter the data along many attributes, e.g. if you only want to download Franka robot data or only data with wrist cams, natural language instructions etc! Tailor the data to your use case!
9/
Thrilled to announce OpenVLA () – a vision-language-action policy for robotic control!
Shout out to my co-leads
@moo_jin_kim
&
@KarlPertsch
; see their threads for overviews of our work.
Here though, I want to talk about observations & next steps! 🧵⬇️
How to use it?
It’s all on HuggingFace — two lines to load the model, no code install needed. We also open-source our full PyTorch training code & data. Scales from fine-tuning on 1 GPU to training billion-parameter VLAs on distributed clusters!
5/
Excited to present SPiRL in contributed talks at the Deep RL and Robot Learning workshops
@NeurIPSConf
! Join us during the poster sessions to chat about all things skill learning & transfer!
DRL Poster: Room F, A1
Robot Learning Poster: C3
w/
@YoungwoonLee
&
@JosephLim_AI
How can we use large offline datasets for accelerating the learning of new tasks? We can transfer skills!
Check out our
#CoRL2020
paper on efficient skill transfer with learned skill priors!
📄Paper:
💻Website & Code:
Thread👇(1/8)
How does it work?
We take a strong open-source VLM, Prismatic 7B, and fine-tune it to predict robot actions, using a curated dataset of 970k robot demonstrations.
This recipe scales, and allows robotics to reuse pretrained models from the community (SigLIP, DinoV2, Llama2) 🚀
2/
Last but not least: Octo is your one-stop-shop for training on OpenX data! We’re releasing high-quality data loaders that work with PyTorch and JAX + a curated dataset split!
7/
Very excited to release the Open X-Embodiment Dataset today — the largest robot dataset to date with 1M+ trajectories! Robotics needs more data & this is a big step!
There’s lots to unpack here, so let’s do a deep dive into the dataset!
🧵1/15
We analyzed the properties of the combined dataset!
First, the number of datasets per robot embodiment: many academic labs use Franka robot arms, so we have many (smaller) Franka datasets and a long-tail of other robot embodiments!
3/
This is great work! 38 fine-tuning tasks for every eval 🤯 thanks for sharing many ablations
@giffmana
and team!
Also confirms our finding that vis encoder fine-tuning is required for finegrained spatial tasks like robot control!
Any plans to release larger PaliGemma models? :)
✨PaliGemma report will hit arxiv tonight.
We tried hard to make it interesting, and not "here model. sota results. kthxbye."
So here's some of the many interesting ablations we did, check the paper tomorrow for more!
🧶
Using the data is easy! All data is stored in tfrecords & we made a colab for visualizing & downloading the data (w/ examples for efficient data loaders)! Each dataset stores observations/actions in its “native” format & resolution, but it's easy to align&mix them on-the-fly!
8/
We plan to expand the dataset over time and e.g. add more mobile manipulation and simulation data.
If you have data that would be good to integrate, simulated or real, please fill out the form:
We’re fully open-sourcing model checkpoints, our pre-training and finetuning pipelines!
Initially, Octo comes in two sizes: Octo-Small (27M params) and Octo-Base (93M params). All models are on HuggingFace, so loading an Octo model is as easy as this:
5/
Big shoutout to Xuanlin (
@XuanlinLi2
), Kyle (
@kylehkhsu
) and Jiayuan (
@Jiayuan_Gu
) for leading this project in a UCSD x Stanford x Google collab! For more details about our approach and results, please check out Kyle’s thread below!
[1/14] Real robot rollouts are the gold standard for evaluating generalist manipulation policies, but is there a less painful way to get good signal for iterating on your design decisions? Let’s take a deep dive on SIMPLER 🧵👇 (or see quoted video)!
We’re hoping to continue this momentum and keep growing the dataset 🚀! We’re still figuring out the details, but if you or your lab have data you’d like to contribute feel free to shoot an email to open-x-embodiment
@googlegroups
.com and we will get back to you! :)
13/
Bonus: with slight tweaks to our model we can make it predict semantic bottlenecks between start and goal. In this case our model learns to predict the subgoal *and* its temporal placement, allowing for non-even splits of the long-horizon problem.
The HuggingFace integration also means that OpenVLA supports all 🤗 magic out of the box, like LoRA fine-tuning, quantized inference etc etc (see paper for detailed analysis of these)!
This makes billion-param models much more accessible in robotics, like it has in NLP!
6/
The distribution of objects is diverse & reflective of objects a robot would encounter "in the wild”, like common furniture pieces, food items, appliances etc. There is still a long way towards real world diversity, but we hope that this dataset can build a good foundation!
7/
In all seriousness though, being able to "program" *and* "debug" your robot in natural language will be tremendously useful when the job of teaching robots new skills is no longer done by machine learning experts in labs but end users in homes!
Great job Lucy!! :)
Please check out Moo Jin’s thread for more details about OpenVLA — Moo Jin really carried the torch in this project, which was the first project in his PhD! Way to go Moo Jin! :)
When collecting your fine-tuning data, start with little variation in terms of objects, positions, scenes, backgrounds, camera angles, etc. It's easier to catch bugs in your robot pipeline this way.
But, for best policy generalization, collect more diverse demo data later!
3/
Jun will present our work on augmenting RL w/ motion planners at
@corl_conf
today. Our RL agents learn to use motion planners for solving challenging manipulation tasks w/ many obstacles!
Interactive Session: today, 11.10am PST.
Led jointly by Jun (
@junjungoal
) &
@YoungwoonLee
.
Octo is only the first step towards building generalist robot policies and we’re planning to improve the models over time — larger sizes, more robot morphologies, RL etc etc — really excited to see how folks will use Octo! :)
8/
Enjoyed reading this paper! Still lots of work to do to get sufficiently diverse task distributions in more realistic domains like robotics, but many of the ideas on ranking multi-task agents & automated task curricula seem generally applicable!
Very excited to release our new work: Open-Ended Learning Leads to Generally Capable Agents. tldr; algorithm that dynamically shapes task distributions to train agents on huge task space, resulting in surprisingly general behaviour
Thread: (1/n)
Joseph will talk about a lot of our skill-based learning works in the PRL workshop
@corl_conf
today!
Starting at 11.30am NZ time (in ~30 mins) — join with this zoom link:
We are presenting our work on keyframe-based prediction and planning at
#L4DC
today! We also made a video, so it will take you only 5mins to get a gist of the paper!👇
Website:
Paper:
Video:
Most importantly: 99% of applications will require *fine-tuning*, i.e. collect a small dataset <100 robot demos in your target domain & fine-tune OpenVLA on it.
Why? OpenVLA needs to learn your robot's action space, camera setup etc. More on 0-shot usage at the end!
2/
Final thought re 0-shot: we want models that can control your robot out-of-the-box, just like downloading an LLM checkpoint.
Current limitation: models are not conditioned on robot URDF / action space definition etc, so they need to learn it implicitly via fine-tuning.
9/
@yuramel2000
@berkeley_ai
You can run inference comfortably on standard consumer GPUs. A 4090 will run our biggest model at ~13it/sec, a 2080Ti still at ~5hz I believe! The smaller model is even faster if you need high control frequency
Finally, we analyzed the distribution of skills & objects in the dataset based on the language annotations (~60% of the datasets have language instructions).
While common pick-place tasks are most frequent, there is a long tail of interesting skills like wiping, assembling etc
6/
@chris_j_paxton
Not sure these scaling laws are meaningful since they aggregate across v/ different tasks. "Curve goes up" is also not surprising since they filtered the raw data for "curve goes up". We need scaling laws but a meta-study is likely not the way to do it w/o agreed upon benchmarks?
How does it perform?
Better than any previous generalist robot policy: we test OpenVLA for controlling multiple robots “out-of-the-box” & outperform our own Octo model across the board.
We also match or outperform RT-2-X, a 55B closed VLA — the key: a larger robot dataset.
3/
@ericjang11
Great video! Specializing the generalist models to shorten finetuning + data iteration cycles is clever! One Q: is the data that finetunes the specialized model well also sufficient to upstream the skill into the generalist? Do you need to collect more of the same "type" of data?
Once trained, you can load your fine-tuned OpenVLA model for inference easily via 🤗 AutoModel.
NVIDIA 4090s offer the best inference speed / $$$.
We also provide code for serving fine-tuned OpenVLA models remotely & query via API, if your fastest GPU isn't near the robot!
6/
If we want to build truly “foundational” models for robotics we need to support the diversity of real robot setups!
Despite the added flexibility, we find Octo's performance to be strong compared to RT-1X and even RT-2X + great during finetuning!
3/
@jeasinema
Agreed! While the initial models we trained are 2D-only, the dataset actually has a lot of multi-view / depth cam data, so lots of opportunity for better policies!
No tactile data yet, but would be great to include going forward! Maybe
@LerrelPinto
can help with that? 😉
👆We also estimated the # of visually distinct scenes per dataset & find that this metric is well distributed across robots, w/ many embodiments contributing a significant fraction of the scene diversity. Ultimately, scene diversity may be more important than trajectory count.
5/
Main technical delta vs December-Octo is that we use GPT to generate paraphrases of all language instructions in our OpenX data mix for better language grounding (thanks
@oier_mees
for leading implementation of this feature!)
We also show Octo finetuning to bimanual ALOHA now!
@SOTA_kke
@XuanlinLi2
@kylehkhsu
@Jiayuan_Gu
If there was bad correlation, it would likely be the other way around: the policy works in real but doesn't in SIMPLER bc of real-to-sim gap. SIMPLER worked well for the policies we tested, but may not always hold eg for policies that are sensitive to visual imperfections.
@RemiCadene
💯 -- Thanks for the shoutout Remi!
Small correction: we don't need to modify the policy at all, we're using open-source checkpoints in our exp, some straight from HuggingFace 😉. We simply make sure that the sim is realistic enough so policies trained on real data work in it!
Lack of scale & diversity in robot datasets is demanding a change towards scalable alternatives- LLMs, Sim2Real, etc.
GenAug presents a powerful recipe for using text2image generative models to demonstrate widespread generalization of robot behaviors to novel scenarios.
🧵(1/N)
So lots of potential for making these models easier to use!
Thanks for sticking around! Check out the OpenVLA announcement below if you missed it & give the model a try!
Huge thanks again to my co-leads
@moo_jin_kim
and
@siddkaramcheti
❤️
Very excited to release OpenVLA today, a 7B parameter open-source vision-language-action model (VLA).
🦾 SoTA generalist policy (better than Octo & RT-2-X)
⚡️ Easy to run & fine-tune on 1 GPU with quantization and LoRA
💻 Open-source PyTorch codebase
🤗 Models on HuggingFace
1/
@GlenBerseth
Jup! We use 128 TPUv4 for ~14 hours for pre-training of Octo-Base, so depending on how many A100 you can use it may take a bit longer but should definitely be possible.
@RemiCadene
Key is that we only modify *the sim environment* to mitigate control&visual gap, the policy is 1:1 the same we are using on the real robot & was trained long before this project even started :)
So in some sense this sim eval is "for free", but requires care in setting up the sim
Before you roll out OpenVLA on your robot, it's very helpful to feed in a few training images & make sure the predicted actions match the training data.
This helps to catch silent bugs in your inference pipeline, one of the most common sources of error!
7/
3 mo. ago we released the Open X-Embodiment dataset, today we’re doing the next step:
Introducing Octo 🐙, a generalist robot policy, trained on 800k robot trajectories, stronger than RT-1X, flexible observation + action spaces, fully open source!
💻:
/🧵