I'm super excited to share AlphaZeroᵈᵇ, a team of diverse
#AlphaZero
agents that collaborate to solve
#Chess
puzzles and demonstrate increased creativity. Check out our paper to learn more!
A quick 🧵(1/n)
We are looking for brilliant and creative candidates with strong programming skills to join us at the Discovery team at
@GoogleDeepMind
🧙
We build AI agents that discover new knowledge using RL, planning and LLMs. DM me if you have specific questions about working with us 🙏
We are looking for brilliant and creative candidates with strong programming skills to join us at the Discovery team at
@GoogleDeepMind
🧙
We are building AI agents that create new knowledge using RL, planning and LLMs in domains like Mathematics, chess and more. Please apply
In our
#Neurips2021
spotlight, we study RL problems where the goal is to minimize a cost over the state occupancy. When this cost is linear, we get the standard RL problem. When it is non-linear, we get apprenticeship learning, pure exploration, diversity and more. [1/7]
Excited to share DOMiNO, a method to discover qualitative-diverse policies using a single latent-conditioned architecture and the "reward is enough" principle.
Read more about it here:
DOMiNO's🍕 in Walker walk:
Super excited to share that our Bootstrapped Meta Learning paper led by
@flennerhag
received an Outstanding Paper Award from
#iclr2022
Better meta learning -> doubled the performance of STACX in Atari to a new SOTA. Come talk with us at the poster session!
What should a meta-learner optimize? What if we make it chase its own future outputs?
Turns out, it can improve meta-optimization, set new SOTAs, and lead to new types of meta-learning.
w. Y. Schroecker,
@tomzhavy
,
@hado
, D. Silver, S. Singh.
🧵👇
A rejection story with a happy end. A paper from my
#Phd
was accepted to
#ICML2021
after 4-5 rejections (I lost count honestly). Each time we had reviewers that liked it and some that didn’t. Believing in it and keep improving it over time eventually got it in. Don’t loose hope!
2/2
#ICML2021
submissions rejected. I would like to thank all parties involved. Also, this marks 3 consecutive years of
#PhD
#fellowship
#rejections
. But hey, who cares, I am a rising 🌟 in EECS 😂 and will rise above this one day✌️Encourage you to share your rejection speech👩🏫
Very excited to share AlphaProof, an agent that self-taught itself Mathematics in Lean and achieved a silver-medal standard in the International Math Olympiad 🥈🥈🥈🥈
@leanprover
is a functional programming language for formal Mathematics and a theorem prover. It enables you to
We’re presenting the first AI to solve International Mathematical Olympiad problems at a silver medalist level.🥈
It combines AlphaProof, a new breakthrough model for formal reasoning, and AlphaGeometry 2, an improved version of our previous system. 🧵
Late on arXiv (oral
@CoLLAs_Conf
):
@jelennal_
who did a fantastic internship with us at the Discovery team
@DeepMind
studies how adding context to meta gradients can help agents to adapt when the environment changes. Thanks for sharing
@_akhaliq
Excited to share DOMiNO, a method to discover qualitative-diverse policies using a single latent-conditioned architecture and the "reward is enough" principle.
Read more about it here:
DOMiNO's🍕 in Walker walk:
Our internship program
@DeepMind
opens for applications today. At the Discovery team we look for brilliant and creative people with a passion for RL to join us. Please apply! and DM me if you have specific questions about working with us 🙋
Excited about meta-gradients (MGs)? join our intern
@jelennal_
tomorrow at
@aloeworkshop
to hear about a systematic study of MGs in non stationary RL environments. We study how adding a context to MGs can assist in adapting to changes and visualise what do these MGs learn.
Excited that our paper "Discovering a set of policies for the worst case reward" was accepted to
#ICLR2021
as a spotlight!
We present a policy iteration algorithm that discovers meaningful behaviours in the no-reward setting by maximising robustness!
Which reward is enough? Is the reward a gradient? Of what?
Come and chat with me tomorrow at the Unsupervised RL workshop poster session on my cherry pie recipe ,
I had a fantastic time talking with
@ecsquendor
,
@ykilcher
and
@RobertTLange
about my deep RL journey, automatic discovery of structure and our recent metagradient work
@deepmind
, you can watch it at
Super proud to share ReLoad (), an RL agent that converges in challenging constrained RL problems. The project was led by our fantastic intern
@ted_moskovitz
at the discovery team last summer. A thread 🧵
Tired of ChatGPT screenshots? Miss the old days of watching RL agents walking around doing weird stuff?
Look no further–I'm excited to share my
@DeepMind
internship project, where we develop a method to stabilize optimization in constrained RL.
Link:
🧵
Training RL agents to satisfy constraints is important for safety and efficiency. But standard methods often oscillate between maximizing reward and satisfying constraints. With
@ted_moskovitz
we propose a solution that converges: Reload. Visit us
@icml
!
What are some of the papers we’ll be talking about?
🔵 Using VLMs to help train embodied agents
🔵 Introducing a new family of recurrent neural networks that perform better on long-term reasoning tasks
🔵 Better training RL algorithms within constraints for AI safety
Excited to be traveling for a conference again, and in particular, to my third
#EWRL
! I will give a talk about Domino , our recent paper on discovering high quality diverse policies. Ping me if you want to chat about diversity, convex RL or anything else:)
Excited to share DOMiNO, a method to discover qualitative-diverse policies using a single latent-conditioned architecture and the "reward is enough" principle.
Read more about it here:
DOMiNO's🍕 in Walker walk:
Excited to share DOMiNO, a method to discover qualitative-diverse policies using a single latent-conditioned architecture and the "reward is enough" principle.
Read more about it here:
DOMiNO's🍕 in Walker walk:
Predicting the value of auxiliary policies, in an off-policy manner, can significantly improve
#DRL
agents (unreal, stacx). In a recent work led by Ray Jiang
@icmlconf
we study how to improve this off-policy learning setup and design empathic td algorithms, suited for DRL agents.
I disagree. Building is not understanding. Computers excel in predicting future from past. They scale with compute and history. We are more predictable than we think. But the future is not fixed. We are creative, diverse, irrational and incomputable. The prize has a long tail.
The case for ambition in artificial intelligence research:
Within your lifetime, AI researchers will understand the principles of intelligence—what it is and how it works—well enough to create beings of far greater intelligence than current humans.
In our
#Neurips2021
spotlight, we study RL problems where the goal is to minimize a cost over the state occupancy. When this cost is linear, we get the standard RL problem. When it is non-linear, we get apprenticeship learning, pure exploration, diversity and more. [1/7]
If you want to hear more about this work, please join us today at 17:00 (GMT) for the poster session and on Thu 11:45 for the spotlight and Q&A.
#ICLR2021
Excited that our paper "Discovering a set of policies for the worst case reward" was accepted to
#ICLR2021
as a spotlight!
We present a policy iteration algorithm that discovers meaningful behaviours in the no-reward setting by maximising robustness!
Computer scientists at Google DeepMind have built a diverse array of AI systems, starting with AlphaZero, that can better solve chess problems. The implications of this work extend beyond games.
@StephenOrnes
reports:
Finally on my way to
#NeurIPS2022
! Pls reach out if you want to chat on anything RL (discovery, creativity, adaptivity, diversity or convex utilities) or if you want to hang out around Frenchman st 🥳
I had a great time chatting with
@ron_itelman
about diversity and creativity in AI. You can read more about this work in our recent AlphaZeroᵈᵇ paper: !
It was an honor and a privilege to have the esteemed
@TZahavy
, staff research scientist at
@GoogleDeepMind
, to share his research in creativity and problem solving, on Episode 11 of Principles of Designing Intelligence.
Working on self-tuning agents like STACX and BMG, I was curious to understand what can agents benefit from adapting their hyper parameters while (meta) learning. In this brilliant paper led by
@khimya
we give one answer using an "ARUBA style" analysis of meta-RL. Details in the🧵
Happy to share our work **POMRL: No-Regret Learning-to-Plan with Increasing Horizons** at the intersection of online meta learning and planning in reinforcement learning jointly with 🐘vernadec
@mastodon
.social…
How to overcome short-sightedness⌛️& ill-conditioned outer objectives in meta-learning❓BMG constructs a bootstrap target by updating the learner for a couple more steps 👣 Provides a new ATARI SOTA 🕹️ & opens up new perspectives 🎨
#mlcollage
[35/52]
📜:
Aloha 👋
I am in Honolulu for
#ICML2023
and it will be great to meet old and new friends! Give me a ping and lets meet for coffee or chat. I am interested in anything RL, and more recently in general utilities, diversity and bounded rationality.
Very excited about this work!
We present an algorithm that learns from demonstrations in personalized medicine. The idea is to model each patient as an MDP and generalize to new patients.
We show, in theory & practice, that the right way to do it is via IRL (not BC/IL/AL)!
Interested in
#RealWorld
applications of
#ReinforcementLearning
?
Check out our recent work, “Inverse Reinforcement Learning in Contextual MDPs” where we learn from clinicians how to treat patients with Sepsis!
Paper:
Code:
Check out our recent work led by
@vivek_veeriah
on using meta gradients to
#discover
useful sub goals and options that maximize them.
And thanks
@RobertTLange
for another fantastic summary!
🤖 How can we learn useful temporal abstractions that transfer across tasks? Veeriah et al. (21') propose to discover options by optimizing their parametrization via meta-∇ 🧠 Loved the idea to disentangle option reward & policy.
#mlcollage
[9/52]
📜:
Neural linear bandits combine a dnn with a linear contextual bandit on top of it’s last layer to explore efficiently. In our
@icmlconf
paper led by Ofir Nabati, we address the problem of catastrophic forgetting (due to feature learning), and propose a moment matching solution.
Deep networks combined with finite memory poses a problem for bandit-based algorithms.
In this blog post, Ofir explains their method for overcoming the drift in the learned features and avoiding catastrophic forgetting!
🚀 How can meta-learning, self-attention & JAX power the next generation of Evolutionary Optimizers 🦎?
Excited to share my
@DeepMind
internship project and our
#ICLR2023
paper ‘Discovering Evolution Strategies via Meta-Black-Box Optimization’ 🎉
📜:
Our experiments show that players in AlphaZeroᵈᵇ play chess differently, solve more puzzles together, outperform a more homogeneous team and specialize in different openings. Remarkably, some of the players solve the challenging
@penrose
positions. (5/n)
Another virtual conference is around the corner, so its a good time to thank the organizers and also everyone who came to talk with me!
I enjoyed the discussions, suggestions and constructive feedback a lot this time, this is what conferences are for!
#ICLR2021
FunSearch is one of these projects that really surprised me when I first heard about it, and even impressed me more as I further learned about it.
Huge congrats to the team!
Introducing FunSearch in
@Nature
: a method using large language models to search for new solutions in mathematics & computer science. 🔍
It pairs the creativity of an LLM with an automated evaluator to guard against hallucinations and incorrect ideas. 🧵
Yes! 🧙
AlphaZeroᵈᵇ discovers a group of strong chess players with distinct playing styles. It models the players with a single, player-conditioned architecture and encourages them to play differently using behavioural and response diversity techniques. (4/n)
We therefore take inspiration from humans, who are also computationally bounded, to improve AI performance. In “The Diversity Bonus”,
@Scott_E_Page
observed that diverse teams of thinkers outperform homogeneous teams in complex tasks. Can we bring these ideas to AI? (3/n)
This team have a fantastic ability to summarize knowledge, ask challenging questions and produce high quality content. They are doing a great service to the ML community so make sure you follow them!
♗♞ Chess is a complex game with many possible moves. Even though modern chess programs can play superhuman chess, they have their blind spots. For examples, check out this article
by
@Sam_Copeland
, and read our paper to learn more! (2/n)
Our internship program
@DeepMind
opens for applications today. At the Discovery team we look for brilliant and creative people with a passion for RL to join us. Please apply! and DM me if you have specific questions about working with us 🙋
Are data scientists the new rockstars?
I am recently being harassed by a bot pretending to be “a single women looking to meet people”. It uses the same font in bio + deep fake images & videos generated from the same network.
@Twitter
please help
Our reward has three components. (1) Extrinsic reward. (2) The gradient of a diversity objective, defined as a non-linear function of the state occupancies (Hausdorff distance, Van Der Waals). (3) A multi-objective mechanism to combine the rewards.
DOMiNO's🍕 in Walker stand:
Using JAX’s auto-diff abilities together with its RL libraries made meta gradient research much easier for me. I Highly recommend the JAX ecosystem to everyone!
In a new blog post,
@davidmbudden
and
@matteohessel
discuss how JAX has helped accelerate our mission, and describe an ecosystem of open source libraries that have been developed to make JAX even better for machine learning researchers everywhere:
In a new paper, , we study the projection method of Abbeel and Ng (2004) for Apprenticeship Learning. We show that it is, in fact, an instantiation of the Frank-Wolfe method -- a projection free method for convex optimization. (1/5)
AI ∩ chess twitter: I found this method peculiar and decided to self-participate. Its suggests overfitting to puzzles by repeatedly solving them. Do we humans not overfit? Or is there something special about puzzles that makes them hard to overfit on? 🤔
@ylecun
@HazanPrinceton
In its dual formulation, RL is bilinear in the reward and the state occupancy, hence, the reward is indeed its (negative) gradient w.r.t the state occupancy.
One example is DOMINO (): an RL agent that discovers high quality and diverse policies by maximizing diversity under the constraint of being nearly optimal and demonstrates robustness to perturbed environments. 3/n
On a personal note, I've been working on this problem for the last few years, focusing mainly on apprenticeship learning (, ). I am very excited that we were able to generalize this framework to cover many important problems [7/7].
But gradient ascent descent only guarantees that the average (over training) of the policies converges, and it is impossible to guarantee that the actual policy is good at any point in time! 4/n
@CsabaSzepesvari
@peter_richtarik
Fight gate keeping - when a reviewer targets a specific paper from conference to conference. This is something that is happening and we need to monitor and stop it.
Just finished with my ICML submissions and on my way to AAAI & ALT !!! Starting the week at the GenPlan workshop. We present a work on Inverse RL in contextual MDPs, where we show, theoretically and empirically, zero shot transfer.
We then show how to reformulate the problem as a min-max game involving policy and cost (negative reward) `players', using Fenchel duality. We propose a meta-algorithm for solving this problem and show that it unifies many existing algorithms in the literature. [3/7]
We then show that for a specific instance of the meta algorithm, the non stationary reward is simply the gradient of the cost with respect to the state occupancy of the previous policies. [5/7]
Carmelo Anthony — 37 years old
LeBron James – 36 years old
Trevor Ariza – 36 years old
Marc Gasol – 36 years old
Dwight Howard – 35 years old (turning 36 Dec. 8)
Wayne Ellington – 33 years old
Kent Bazemore – 32 years old
Russell Westbrook — 32 years old
@pcastr
@JohnCLangford
On a similar fashion, one thing that I found really nice in
#UAI
was the discussant format. An author from each accepted paper was requested to prepare a slide with a question / discussion about another paper, which really improved the Q&A.
We demonstrate that our QD policies are robust to environment perturbations, by training DOMiNO in a "baseline" domain and highlighting the QD policies that are robust to domain perturbations.
Confused? consider a simple simple saddle point problem. The dynamics of gradient ascent descent (orange) spirals away from the optimal point, but an OPTIMISTIC version of the gradient bends inside and converges 🧙🧙♂️🧙♀️ 5/n