Review paper on Foundation Models for Decision Making:
Foundation models can characterize various components of decision making, such as states (S), behaviors (A), dynamics (T), task specifiers (R), through generative modeling or representation learning.
Checkout UniMat -- a unified representation of materials that enables scaling of diffusion models to millions of stable crystal structures.
Website:
Paper:
Consider joining our team at Google DeepMind to work on foundation models for decision making, e.g., foundation model alignment, reasoning, planning, simulation, and optimization with foundation models.
Our team (w/Dale,
@daibond_alpha
,
@mengjiao_yang
+ others) at Google DeepMind is looking to hire. If you are interested in foundation models+decision making, and making real-world impact through Gemini and cloud solutions, please consider applying through
Video generation will revolutionize decision making in the physical world like how language models have changed the digital world.
Interested in the implications of video generation models like UniSim and Sora? Check out our position paper:
Happy to share that UniSim was selected for an Outstanding Paper Award at
#ICLR2024
.
Check out the oral presentation today at 10:30am Oral 1B and poster on Wed at 4:30-6:30pm
#87
.
Thanks to the award committee
@eunsolc
,
@katjahofmann
,
@liu_mingyu
,
What does "Learn principles, not formulas. Understand, do not memorize” mean for autonomous agents?
Chain of Thought Imitation with Procedure Cloning!
ArXiv
Code
Site
w/ Dale
@pabbeel
@ofirnachum
As video foundation models reach billions of parameters, how to adapt them to task-specific settings (e.g., animation, robotics) without access to the model weights becomes a pressing issue.
We introduce Video Adapter:
See you all at the 2nd Foundation Models for Decision Making
#NeurIPS2023
workshop tomorrow (Friday) at Hall E2 starting 8:15am. Don't miss out on an exciting line of speakers! See schedule and zoom link at .
Looking forward to presenting the following papers
@icmlconf
:
- Position paper on Video Generation for Decision Making (Tue 1:30 - 3pm
#2613
).
- Code as Reward for real-world RL with VLMs (Thur 1:30-3pm
#1115
).
I talked about our position paper on "Video as the New Language for Real-World Decision Making" on the TWIML podcast
@twimlai
. Check out the conversation below:
Today we’re joined by
@mengjiao_yang
, a senior research scientist at Google DeepMind, to learn why video data offers a better foundation than natural language for teaching AI to understand the world.
🎧 / 🎥To listen to the audio version, visit .
Paper with
@ofirnachum
-- Representation Matters: Offline Pretraining for Sequential Decision Making -- is accepted to
#ICML2021
!
Contrastive pretraining yields huge gains in low-data imitation learning, offline RL, and online RL.
Both RT-Sketch () and RT-trajectory () can be viewed as a form of "chain-of-thought" of agents (). Learning mappings from high-dim images to low-dim controls is difficult, and intermediate info could help.
Heading to
#NeurIPS2023
with an exciting agenda:
[1/5] I will showcase UniSim at the Google DeepMind booth (Hall C, 315) on Mon Dec 11 12-2pm CT. Please stop by to interact with the real-world simulator.
TRAIL's latent action pretraining provably accelerates downstream imitation learning even when the offline dataset is highly suboptimal (e.g., collected from a random policy).
Paper:
Code:
Website:
How can we leverage existing behavior datasets to learn an "easier" action space for control? Many have thought about this question (eg see my OPAL paper w/ A. Ajay), but existing work relies on dataset to already contain good and temporally-extended behavior. And so... TRAIL!
Checkout our ICML paper -- Code as Reward .
VLMs can generate code to compute reward from images, enabling RL agents to learn in the real-world without hand-designed simulators. This compliments generative world models like UniSim .
We are excited to announce that our work "Code as Reward: Empowering Reinforcement Learning with VLMs" was accepted to ICML 2024. This work was done with Sami Nur Islam
@MartinKlissarov
Doina Precup
@mengjiao_yang
@ankit_s_anand
.
Despite all the buzz about decision transformers, it's well-known they can be unboundedly suboptimal in stochastic envs. The issue is due to conditioning the policy on return, a highly stochastic quantity. We introduce "dichotomy of control" to solve this:
CALM () and CHAI () are our first attempt at combining offline RL with language models to solve task-oriented dialogue. While dialogue is hard, our initial set of results are highly encouraging:
Offline RL is a natural fit for dialogue: RL with humans is hard, but data of humans talking to humans is plentiful. In two new papers, we explore offline RL and for end-to-end dialogue systems with Transformers!
CALM:
CHAI:
🧵->
I'll present Unified Materials (UniMat) today 10:45am - 12:45pm at poster
#170
#ICLR2024
.
Come and learn about the initial effort from
@GoogleDeepMind
on developing generative models for materials.
Joint work with KwangHwan Cho,
@amilmerchant
,
@pabbeel
, Dale Schuurmans,
Checkout UniMat -- a unified representation of materials that enables scaling of diffusion models to millions of stable crystal structures.
Website:
Paper:
Check out Video Language Planning!
VLP uses vision-language models as policy and reward, and UniSim as dynamics.
Foundation agent models can now conduct search and planning in foundation world models.
Introducing Video Language Planning!
By planning across the space of generated videos/language, we can synthesize long-horizon video plans and solve much longer horizon tasks than existing baseline (such as RT-2 and PALM-E).
(1/5)
I'll present Video Adapter today (Friday) 10:45am - 12:45pm at poster
#200
#ICLR2024
.
Come and learn about how we can adapt large video foundation models like UniSim to domain-specific settings (e.g., animation, robotics) without requiring access to the foundation model
As video foundation models reach billions of parameters, how to adapt them to task-specific settings (e.g., animation, robotics) without access to the model weights becomes a pressing issue.
We introduce Video Adapter:
Off-policy evaluation is not the full story - we introduce bayesian decision making to select policies based on any criteria:
Offline Policy Selection under Uncertainty:
code:
with
@daibond_alpha
,
@ofirnachum
,
@georgejtucker
, Dale
New paper: Energy-Based Processes for Exchangeable Data.
We combine EBMs with stochastic processes to model set structures with arbitrary cardinality.
Paper:
Code:
Joint work with
@daibond_alpha
,
@hanjundai
, and Dale Schuurmans.
UniSim is trained on broad data rich in different axes (internet text-image, human activity videos, simulated executions, real robot videos, panorama scans, etc).
In our recent work ALPT, I was surprised that inverse dynamics model pretrained on Freeway (Atari game only moving up and down) dramatically improves performance of Breakout (only moving left and right). One step closer towards cross-domain generalization:
I am amazed at the generalization performance of inverse dynamics models for ALPT, even winning out in situations where source/target tasks have completely disjoint action spaces! Awesome work with
@DavidAVenuto
@mengjiao_yang
@pabbeel
@IMordatch
, Precup.
See you all at the 2nd Foundation Models for Decision Making
#NeurIPS2023
workshop tomorrow (Friday) at Hall E2 starting 8:15am. Don't miss out on an exciting line of speakers! See schedule and zoom link at .
I will dive into more detail of how we have been leveraging UniSim to solve embodied decision making tasks at the Generative AI for Decision Making workshop today 4pm-4:30pm (Lehar 3).
How can we effectively train generalist multi-environment agents? We trained a single Decision Transformer model to play many Atari games simultaneously and compared it to alternative approaches:
Curious about how to make OPE *practical* for selecting policies in practice? Checkout our
#AISTATS2022
poster on Wed 3/30 at 8:30am PT (session 5).
Offline Policy Selection under Uncertainty
w/
@daibond_alpha
,
@ofirnachum
,
@georgejtucker
, Dale
In our recent work ALPT, I was surprised that inverse dynamics model pretrained on Freeway (Atari game only moving up and down) dramatically improves performance of Breakout (only moving left and right). One step closer towards cross-domain generalization:
I'll present Dichotomy of Control today at 11am in the auditorium (Oral 3 Track 1) and poster session 11:30am - 1:30pm in MH1-2-3-4,
#119
. Hope to see you there.
#ICLR23
Recording:
Despite all the buzz about decision transformers, it's well-known they can be unboundedly suboptimal in stochastic envs. The issue is due to conditioning the policy on return, a highly stochastic quantity. We introduce "dichotomy of control" to solve this:
Foundation Models for Decision Making (FMDM) workshop
@NeurIPSConf
is open for submissions! Remember to submit your work (4-9 pages) before Sept 22 at .
We are open for submissions!
I know there are lots of people working on large models, pretraining, cross-domain/agent generalization for RL. Please submit your papers to the 1st FMDM workshop at NeurIPS 2022!
For instance, skill discovery and Decision Transformer are examples of generative models of behavior, whereas model-based offline RL and Trajectory Transformer are examples of generative models of the world dynamics.
TRAIL's latent action pretraining provably accelerates downstream imitation learning even when the offline dataset is highly suboptimal (e.g., collected from a random policy).
Paper:
Code:
Website:
Large language models can serve as agents or environments, supporting interactions with humans, tools, and the real world, enabling new learning environments such as the internet and human knowledge.
Aside from directly solving tasks, video generation is also a realistic simulator for complex games (), which can be combined with model-based planning, or be used to create new games ().
I am really excited to reveal what
@GoogleDeepMind
's Open Endedness Team has been up to 🚀. We introduce Genie 🧞, a foundation world model trained exclusively from Internet videos that can generate an endless variety of action-controllable 2D worlds given image prompts.
Large pretrained vision / language foundation models can characterize various perceptual components of decision making agents such as image observations, language actions, and language goals (e.g., SayCan, PALM-E) through plug-and-play.
Super exciting work that connects contrastive pretraining to learning representations of transitions/rewards in a dynamical system, which probably improves downstream behavioral cloning tasks!
For those who want a distraction from their submissions... checkout our new paper! w
@mengjiao_yang
about *provable* bounds of representation learning for downstream imitation (ie behavioral cloning, ie max-likelihood). Lots results & insights in this paper I'm excited about...1/
UniPi came about as a part of our broader effort on *Foundation Models for Decision Making*, where vision-language architectures and pretrained models are applied to decision making. Check out other exciting work in this area:
Our unified DICE framework reveals that unlike the Q-values, the dual estimates of policy value in terms of state-action distributions offer greater flexibility in regularization while being unbiased, and are robust to scaling and shifting of MDP rewards.
Policy evaluation via duality/Lagrangian methods presents a lot of choices (how to setup the LPs, regularize them, etc). In we examine how these choices affect accuracy of final eval. Lots of insights in this paper, many of which I didn't expect....
Evaluation on simulated robotic navigation, manipulation, and game environments show that procedure cloning exhibits significant generalization to unseen environment configurations such as maze layouts, positions of objects, transition stochasticity, and game difficulties.
Video Adapter works by composing scores of a pretrained text-to-video model with scores of a domain-specific small model (with 1% parameters) during sampling time, achieving high-quality yet flexible video synthesis without requiring gradient updates on the pretrained model.
One key to the success of foundation models is a unified representation. Language uses tokens. Vision uses pixels. What about materials?
We develop a unified way to represent materials by storing the locations of atoms to their respective entries in the periodic table.
Lastly, we note the connection between "chain of thought imitation" and "chain of thought prompting" by
@_jasonwei
et al. --- decomposing multi-step problems into intermediate steps and learning the intermediate steps using a sequence model is applicable to a variety of problems.
With this unified representation, we train a diffusion model to generate materials by moving atoms from random locations back to their original locations. Atoms that do not exist will be moved to a special null location. This allows generation of crystals with arbitrary elements.
I found UniPi’s *video-as-policy* and *text-as-task* abstractions to be empowering. These abstractions unify environments with different state and action spaces and enable policy learning across broad datasets.
@xiao_ted
An interesting next step would be to elicit these useful intermediate information in the agent automatically, so that humans don't have to draw goals / motion plans during inference, similar to how LLMs can automatically output intermediate reasoning steps.
How far can we go with vision alone?
Excited to reveal our Large Vision Model! Trained with 420B tokens, effective scalability, and enabling new avenues in vision tasks! (1/N)
Kudos to
@younggeng
@Karttikeya_m
@_amirbar
,
@YuilleAlan
Trevor Darrell
@JitendraMalikCV
Alyosha Efros!
Similar to text, video is a unified interface that can absorb internet knowledge and represent diverse tasks. This allows us to pour internet data into a single model that can solve many tasks through conditional generation.
@danijarh
Thanks Danijar! Depending on the number of denoising steps, generating 16 frames could take anywhere between 1-20 seconds. We found for RL / planning on Language Table, having very few denoising steps (e.g., 8) is sufficient, which takes roughly a second.
Generative video simulator is also useful for optimizing control input in science and engineering domains where abundant video data can be collected but the underlying physical dynamics are hard to be explicitly expressed (e.g., cloud movement, interaction with soft objects).
We propose procedure cloning (an alternative to BC), which applies supervised sequence prediction to imitate the series of expert computations. Procedure cloning learns not only what to do (i.e., the output action), but how and why to do it (i.e., the procedure).
Glad to see our initial work, Attentive Contrastive Learning (BERT for RL) in 2021 , has been joined by many subsequent work in applying BERT-style autoencoding objectives to sequential decision making (e.g., MaskDP
@fangchenliu_
and MTM
@philippswu
).
We scale UniMat to train on >2M low-energy materials, and show that conditional generation (conditioned on atom type) can generalize to generating more difficult structures, outperforming random structure search (the current leading method) in discovering new stable materials.
Before developing better dialogue agents, we need reliable and scalable evaluation of dialogue systems. ENIGMA evaluates dialogue agents by estimating how a human user would rate an agent via off-policy evaluation with DICE (distribution correction estimation).
An ideal environment for evaluating dialog agents, i.e., the Turing test, needs to involve human interaction, which is not affordable for large-scale experiments. Our EMNLP 2021 paper proposes a new framework – ENIGMA for automating the Turing test. (1/4)
Under DFT evaluations. UniMat generates materials with significantly lower decomposition energy and many more new stable materials (with respect to Materials Project 2021) compared to previous state-of-the-art generative models.
Video Adapter potential has broad applications such as anime production and domain randomization for bridging the sim-to-real gap in robotics. Great collaboration with
@du_yilun
,
@daibond_alpha
, Dale, Josh, and
@pabbeel
.
Video is also a unified observation space across different embodiments, so we can generate visual execution plans for different robots using a single video generation model:
@jeasinema
Hi Xiaojian, thanks for the interest! For 1), we didn't train separate models for each domain; we used a single conditional video diffusion model for all the data mixtures (the mixture ratio is in Appendix B of the paper).
We formulate the “chain of thought” imitation learning problem, where an agent also has access to the intermediate computations that generated the expert state-action pairs, such as planning, search, or some other multi-step algorithm (e.g., BFS, MCTS).
A model can answer people’s questions by generating how-to videos (e.g., “how to make a sushi roll”), which may be more preferable than textual responses.
Morning speakers:
10-10:30 Gato: A Generalist Agent
@gbarthmaron
10:30-11 Open-Ended Embodied Agents with Internet-Scale Knowledge
@DrJimFan
11-11:30 What does an intelligent robot need to know? (Leslie)
11:30-12 Learning and Leveraging Foundation Models in Robotics
@DorsaSadigh
Video Adapter achieves much better FVD, FID, and Inception Scores than both the pretrained and the task-specific model, and outperforms finetuning under the same TPU hours.
What does "Learn principles, not formulas. Understand, do not memorize” mean for autonomous agents?
Chain of Thought Imitation with Procedure Cloning!
ArXiv
Code
Site
w/ Dale
@pabbeel
@ofirnachum
Our breadth study shows that a class of contrastive self-prediction loss works particularly well while many other objectives perform poorly. Our depth study ablates over action/reward prediction/reconstruction, momentum network, tandem training, etc to illustrate their effects.
@itif_workshop
[5/5] We will be hosting the 2nd workshop on Foundation Models for Decision Making on Friday Dec 15 starting 8:15am CT in Hall E2. Don't miss out on this exciting workshop!
Unfortunately, I will have to miss the oral presentation.
@du_yilun
will talk about the work instead. But hope to see you at the poster session tomorrow!
P.S. Lesson learned -- apply for Schengen visa early; appointments in certain regions fill up two months in advance!