📣 New paper! Open-Endedness is Essential for Artificial Superhuman Intelligence 🚀 (co-lead
@MichaelD1729
).
🔎 We propose a simple, formal definition for Open-Endedness to inspire rapid and safe progress towards ASI via Foundation Models.
📖
🧵[1/N]
Absolutely delighted that our paper came out in Nature Communications today. This was an incredible effort from an amazingly talented team at GDM that I had the great privilege to lead.
Perhaps the most innovative work that I've had the pleasure of being a co-author on. You've heard of generative video (at scale) but what about generative simulation at YouTube scale?!?!
Google presents Genie
Generative Interactive Environments
introduce Genie, the first generative interactive environment trained in an unsupervised manner from unlabelled Internet videos. The model can be prompted to generate an endless variety of action-controllable virtual
I’m super excited to share our work on AdA: An Adaptive Agent capable of hypothesis-driven exploration which solves challenging unseen tasks with just a handful of experience, at a similar timescale to humans.
See the thread for more details 👇 [1/N]
Enormously excited to co-author this incredible paper. It represents the latest and most exciting step towards generating cultural evolution in AI - a personal mission of mine for the past 7 years! Kudos to
@JonnyCoook
and
@_chris_lu_
who did all the heavy lifting. [1/n]
1/ 🚀 Presenting AGI - Artificial Generational Intelligence 🚀
We apply the concept of cultural accumulation to RL and find that agents can improve across generations, outperforming those trained for a single lifetime of the same experience budget!
Co-led w/
@_chris_lu_
.
🧵
My favourite poster of the week
@icmlconf
: like Voyager (+ MCTS) but for social deduction games. Really exciting work in the burgeoning field of Open-Ended Cooperative AI by Jonathan Light et al.
Cool prize: ! The problems remind me of the “production rules” from our AdA paper (). I wonder whether large-scale meta-RL with an LLM-powered curriculum might be a quick(ish) way to win $1M 🌶️?
We are heading for a paradigm shift in the process of scientific discovery itself. What an incredible privilege it is to work on open-endedness at this moment. Huge congratulations to my friends and colleagues in the field for this leap towards general AI-accelerated science!
Excited to share The AI Scientist! We use LLMs to autonomously come up with research ideas, implement them, do literature search, write them up, and review them -- producing full-length papers on AI without human intervention.
Co-led with
@cong_ml
and
@RobertTLange
Yet more evidence that data-focussed research is the path to victory! Teaching new in-context abilities is typically bottlenecked by the data distribution. Here the authors prove that point for search.
Language models struggle to search, not due to an architecture problem, but a data one! They rarely see how to search or backtrack. We show how LLMs can be taught to search by representing the process of search in language as a flattened string, a stream of search (SoS)!
Awesome work: enormously exciting to see a reimplementation of XLand which is so efficient and accessible. Here's to a bright future for meta-RL research!
🔥 Imagine if you could train Meta-RL agents for 1 TRILLION transitions under 40 hours?
We present XLand-MiniGrid — JAX-accelerated meta-reinforcement learning environments inspired by XLand (
@FeryalMP
) and MiniGrid (
@Love2Code
).
code:
I’m pleased (and unsurprised) that generating synthetic data makes a lot of progress on ARC. I don’t view this as a weakness of the challenge, however - rather as an endorsement of the power of open-ended program-based data generation for model improvement!
ARC-AGI’s been hyped over the last week as a benchmark that LLMs can’t solve. This claim triggered my dear coworker Ryan Greenblatt so he spent the last week trying to solve it with LLMs. Ryan gets 71% accuracy on a set of examples where humans get 85%; this is SOTA.
I don't know whether this five-tier system is genuine. Nevertheless, it is interesting (and unsurprising to me) that the Levels get progressively more Open-Ended and Multi-Agent. Perhaps
@sama
follows my external talks...
OpenAI reportedly internally introduced a new five-tier system to track its progress toward AGI.
The classification system ranges from Level 1 (current conversational AI) to Level 5 (AI capable of running entire organizations).
The field of cultural evolution in AI is continuing to grow! Inspiring to see this early stage report. Adding a "related work" section would help drive the cultural evolution of the research community.
Very proud to have played a small role in this awesome team! It's almost inconceivable to me that 15 years since I was trying (and failing badly) to solve BMO1 problems, we have now made an AI system that is nearly gold-medal standard on this year's IMO 🍾🤯
We’re presenting the first AI to solve International Mathematical Olympiad problems at a silver medalist level.🥈
It combines AlphaProof, a new breakthrough model for formal reasoning, and AlphaGeometry 2, an improved version of our previous system. 🧵
Super chuffed to have been a member of the Genie team which won a best paper award today! Big props to
@ashrewards
and
@jparkerholder
for an awesome oral talk describing the work 🧞♂️🍾
Come and see
@_chris_lu_
and myself present the fascinating Artificial Generational Intelligence poster right now in Hall A2
@icmlconf
. Bravo to
@JonnyCoook
who led this work!
We are at a moment of maximum leverage for Cooperative AI. The theory and organisational infrastructure is well-developed, and multi-human multi-AI systems are nascent. Now is the best time to develop AI systems and governance protocols towards societal good.
Epicly good work from some good friends who are very talented scientists! Self-improvement in cooperative games (like language) requires self-generation of diversity, unlike for zero-sum games. This paper paves the way for continuous, open-ended discovery (and patching) of safety
Meta presents Rainbow Teaming
Open-Ended Generation of Diverse Adversarial Prompts
As large language models (LLMs) become increasingly prevalent across many real-world applications, understanding and enhancing their robustness to user inputs is of paramount importance. Existing
Pre-learning best responses is not enough: safe, efficient autonomous vehicles need to adapt online based on inferences about others' policies. A beautiful real-world illustration of the problem!
Waymo car failing to coordinate w/ another Waymo (credits in the comment). Interesting to see a toy example from my grant applications play out in the real world. Two cars playing a best-response to a human driver model are not mutually compatible, multi-agent challenges are real
Very excited to land in Chicago to lecture on Open-Ended Cooperative AI at the
@SIOEcon
AI Bootcamp tomorrow. There are a wealth of possibilities for interdisciplinary collaboration in this field, and I’ll learn as much as I teach!
What an extraordinary week in Santa Cruz for the
@coop_ai
Retreat and Summer School. It has been such a privilege to talk with so many deep thinkers and to have the opportunity to inspire the next generation of brilliant students. The future of the field is bright!
Many congratulations Sir
@demishassabis
on a hugely well deserved Knighthood. Thanks to Demis, I have spent the last 7 years living my childhood dream of building intelligent machines, amidst the pioneering and deeply kind culture at
@GoogleDeepMind
, and this is just the start!
Sometimes, when you've been running, you have to walk for a bit before you can carry on running. And that walk gives you perspective to better decide where to run next. Innovation thrives when you can optimally alternate space and pace.
Increasingly I find that I skim many papers rather than reading fewer in detail. I find that this practice annoys me. Therefore for the next week, I am going to read one paper per day thoroughly and share some thoughts.
Is Open-Endedness essential for ASI? Come see our
@icmlconf
poster in Hall C 4-9
#613
at 1:30pm and our oral at 4:30pm in Hall C 1-3. Can't wait to answer some probing questions and to debate our position with you!
Today,
@edwardfhughes
and
@MichaelD1729
from
@GoogleDeepMind
's Open-Endedness Team will be presenting "Open-Endedness is Essential for Artificial Superhuman Intelligence" as an Oral at 4:30pm in Hall C1-3.
A great pleasure to give a lit review lecture on
#OpenEndedness
@imperialcollege
this afternoon. Many thanks to Anastasia Borovykh for the invitation. By chance, I covered many of the same themes as in
@DrJimFan
's excellent Nvidia keynote (minus the humanoids!)
Do check out the paper on arXiv and keep an eye out for exciting follow-ups from
@j_foerst
's world leading Oxford lab, and our open-endedness community at
@GoogleDeepMind
! I'll be at the
@coop_ai
Summer School in June and at ICML in July if you want to ask questions in person!
We are happy to present "Meta-Learning an Evolvable Developmental Encoding"! 🧬
Generative models can work as learnable representations for blackbox optimization but they are not designed to be easily searchable.
We present a system that can meta-learn such representation by
The best way to (re)build your confidence as a scientist is to immerse yourself in the small stuff: engineering a feature, writing a 1 paragraph conjecture, reviewing a paper. Eventually you realise (remind yourself) that the small stuff *is* the science.
❤️ A huge shout out to my wise, patient, imaginative and dedicated co-authors
@jparkerholder
,
@FeryalMP
,
@aditimavalankar
,
@YugeTen
, Tom Schaul,
@_rockt
.
🤝We found that this paper helped us align our shared interests and hope it is similarly useful for the community.
🧵[7/N]
Should AI agents write their own sandbox code? Permitting this feels eminently open to backdoors. Yet preventing it may lead to unexpected exploits. After all, humanity evolves its own “sandbox code” in the form of norms and institutions, reducing brittleness.
Applications for the 2024 Cooperative AI Summer School are now open! June 19-23, Santa Cruz, CA. Confirmed speakers include
@edwardfhughes
,
@polynoamial
,
@fangf07
,
@nsrg_shah
, and Joe Halpern. Find out more and apply via our website: .
Excited to share my first work from my internship
@SakanaAILabs
!
We used LLMs to design and implement new preference optimization algorithms for training LLMs, discovering cutting-edge methods!
Co-led with
@samianholt
and Claudio Fanconi.
Details in thread 🧵 (1/N)
Awesome to finally meet you in person
@jeffclune
, after so many years being inspired by your work! There has never been a more exciting time to do Open-Endedness research: it's a privilege to work in such a creative and game-changing field.
It was magical to return to Oxford to give a talk, seeing old friends and making new ones. That's especially true because Jakob
@j_foerst
is a great host that really knows how to roll out the red carpet!
Super excited to see
@_rockt
’s book in the pipeline! He is one of the most lucid, knowledgeable and engaging thinkers I have ever worked with, and I’m delighted that a wide audience can now benefit from his insights. The perfect stocking filler!
Excited to contribute to
@SevenDialsBooks
' popular science book series "10 Things You Should Know" with a book on AI.
AI will be humanity's most transformative technology. In ten short and easy to digest essays written for the general public, I am giving an overview of what AI
🌐 Open-endedness can be supercharged by "standing on the shoulders of giant human data" in the words of
@jennyzhangzt
&
@jeffclune
.
🤖 Foundation models provide the knowledge to guide open-endedness towards artifacts that are interesting, novel and useful to humans.
🧵[4/N]
Awesome work on cultural evolution among LLMs! This will only become more relevant as advanced AI becomes embedded in society. We must understand multi-step multi-agent dynamics among LLMs to avoid undesirable attractors and realise the potential for LLM-powered innovation.
What happens when LLMs play the Telephone game? ☎️
In this new preprint, we analyse the evolution of texts as they are transmitted between LLM agents 🤖💬🤖💬🤖💬
Do text properties converge to attractors? 🧲
How is this influenced by the task📝 and model⚙️?
1/13🧵
: an AI-for-proteomics startup that just came out of stealth.
They are announcing ESM3 a 98B-paramter generative LLM for "programming biology."
Using ESM3 and a simulated evolutionary process, they have produced a new type GFP (Green Fluorescent Protein)
I had the pleasure to be an internal reviewer on this excellent work. Very exciting to see self-referential self-improvement have such a significant impact on a wide range of LLM benchmarks. Another wonderful example of open-endedness "going mainstream"! Bravo to the authors!
🔮 Open-endedness has the potential to reach superhuman intelligence. This raises vital ethical and safety considerations.
⚠️ AI systems must remain explainable and controllable: by our definition, AI systems which are incomprehensible are not open-ended.
🧵[5/N]
The surge in
#OpenEndedness
research on arXiv marks a burgeoning interest in the field!
The ascent is largely propelled by the trailblazing contributions of visionaries like
@kenneth0stanley
,
@jeffclune
, and
@joelbot3000
, whose work continues to pave new pathways.
Really pleased to see open-endedness research made more accessible with academic scale compute. A major bottleneck to progress is building a community of great open-endedness researchers across many institutions, and this will go a long way to addressing that challenge. Bravo!
I’m excited to announce Craftax, a new benchmark for open-ended RL!
⚔️ Extends the popular Crafter benchmark with Nethack-like dungeons
⚡Implemented entirely in Jax, achieving speedups of over 100x
1/
It has been a great privilege to think deeply about the Ethics of Advanced AI Assistants with this incredible team of co-authors. For a multi-agent take, jump in at Chapter 14!
1. What are the ethical and societal implications of advanced AI assistants? What might change in a world with more agentic AI?
Our new paper explores these questions:
It’s the result of a one year research collaboration involving 50+ researchers… a🧵
🌍 Models like Gemini are generally capable but don't create new knowledge.
🪄 Advances like AlphaFold have revolutionised their fields but aren't fully general.
♾ How can we define and build AI capable of endless innovation in science and technology?
🧵[2/N]
This is just ridiculous. Will
@Keir_Starmer
do something about this wilful self-sabotage of our research ecosystem, if he is elected? He jolly well should do: innovation is the engine of growth!
New analysis from the
@royalsociety
shows just how high UK visa costs for researchers are compared to upfront costs for similar visa routes in 13 other countries.
Here, we take a look at how the UK compares to countries in the G7 included in the analysis 🔍⤵️ 1/5
There is far too much admin associated with paper publication. Globally we should lobby for changes in legislation to remove the need for unnecessary consent to publish and copyright waivers. Scientists just want their research read and built on!
In this paper, we provide the first proof of concept for cultural evolution among RL agents. Our generational method outperforms "single lifetime" RL both in a few-shot and across training. In other words, culture begets innovation, along the lines of
@jeffclune
's AIGAs! [4/n]
I find NetHack increasingly interesting. Its dynamics are both firmly rooted in human knowledge and require careful experimentation and reasoning. Visually and textually it is in the distribution tail for foundation models, so is a good discriminator for open-ended capabilities.
Happy "AI still can't learn to play NetHack" day for those of you who celebrate.
On this day in 2020, we released
@NetHack_LE
. Despite tremendous progress in AI over the last four years, this challenge is still very far from being solved. From our NeurIPS paper
Excited to have arrived in Vienna for
@icmlconf
. Looking forward to a week of creative discussions, learning and deep thinking with friends new and old!
Really interesting work - we saw in AdA that conditioning on # shots led to more expressive and adaptive policies. Awesome to see this principle lifted to the scale of an entire RL run!
Meta-learning can discover RL algorithms with novel modes of learning, but how can we make them adapt to any training horizon?
Introducing our
#ICLR2024
work on discovering *temporally-aware* RL algorithms!
Work co-led with
@_chris_lu_
, in
@FLAIR_Ox
and
@whi_rl
Came across the
#OpenEndedness
related episode of the
@lexfridman
podcast with
@leecronin
. I particularly enjoyed this quote, which fits in very well with the philosophy of unsupervised environment design: “I’m excited because I think selection isn’t special at all. I think what
Very cool work - exciting to see a new paradigm for search in imperfect information games, addressing the limitations of public belief state algorithms (which had always somehow jarred with me).
SOTA AI for games like poker & Hanabi rely on search methods that don’t scale to games w/ large amounts of hidden information.
In our ICLR paper, we introduce simple search methods that scale to large games & get SOTA for Hanabi w/ 100x less compute. 1/N
Cultural evolution is the fastest known intelligence generating mechanism in the universe, and the driving force behind human skills and technology. Recent work by luminaries like
@JoHenrich
,
@CeliaHeyes
and
@mmuthukrishna
have given us significant insight into this process.
Honestly extremely impressive: (1) the underlying AI research, (2) the imaginative product realisations, (3) the organizational and ecosystem design that has enabled this, (4) the engaging, relatable presentation style.
There are roughly four levels of generalization:
0. No generalization (e.g. a database)
1. Having memorized *the answers* for a static set of tasks and being able to interpolate between them. Most LLM capabilities are at that level.
2. Having encoded generalizable programs
Fast, flexible cultural transmission underpins human intelligence. Our team trains an AI capable of real-time cultural transmission in previously unseen navigation tasks. The agent follows expert demos & reproduces them reliably after the expert leaves: 1/
In previous work in
@NatureComms
() we showed that agents could learn to learn from each other from on-the-fly: just like humans! But can this "social learning" generate an open-ended evolution of ideas in AI, just as it has for humanity? [3/n]
🎉 New paper 🎉
We teach Transformers to do A* search (I had to relearn how A* works). Then, we're curious to see if it can self-improve, and it did surprisingly well. This direction of search, plan, self-improve is very exciting!
🧑 In our definition, an observer judges artifacts to be open-ended when they are both *novel* and *learnable*.
🧬 Artifacts are novel when they are more surprising further into the future. Artifacts are learnable when they are more predictable given more history.
🧵[3/N]
In five years time we will think of foundation models like we think of operating systems: the next phase of transformative innovation will be on the level above.
This is extremely exciting work. For a while I have been concerned that humans adapt to AI much faster than AI adapts to humans. This is the first paper that I have seen which gracefully allows us to take into account that process. Critical for safety!
Excited to share a unifying formalism for the main problem I’ve tackled since starting my PhD! 🎉
Current AI Alignment techniques ignore the fact that human preferences/values can change. What would it take to account for this? 🤔
A thread 🧵⬇️
Point of personal pride ✨: I wrote the gold implementation of the best published model, after several years away from IC coding as as a project lead.
#unleading
ftw!
@francoisfleuret
@ChrSzegedy
Hope that's helpful - and excited about the potential for community building here! It would be really cool to think about combining your approach with some of the ones I've mentioned in this thread.
Late to the party, but the DPO paper is a great example of really impactful theoretical work. It's immensely valuable to find mathematical results that take short-cuts to better performance.
Ever wondered how your LLM splits numbers into tokens? and how that might affect performance? Check out this cool project I did with
@djstrouse
: Tokenization counts: the impact of tokenization on arithmetic in frontier LLMs.
Read on 🔎⏬
We test our agents on several environments, including a variant of the classic Goal Sequence by
@natashajaques
and
@kandouss
, and the Traveling Salesperson Problem. The results are robust, and I fully expect the benefits to scale up with industry compute! [5/n]
@francoisfleuret
@ChrSzegedy
A little further afield, the recent Quality Diversity from Human Feedback work by Li Ding et al. has a cultural evolutionary flavor to it for me.
Wider point: the open-endedness and cultural evolution communities can learn a lot from each other!
Personal reflection: one thing that I noticed in my time on the team was its incredible tenacity, individually and collectively. Most of what I contributed were negative results, and many ideas didn't work out. Sailing on bravely through these choppy waters was vital to success!
Our methods bear a pleasing analogy to knowledge accumulation and skill accumulation in humans. This continues an increasingly rich tradition of building bridges between multi-agent AI and the social sciences, which
@jzl86
and I have collaborated on for many years. [6/n]
True or false: not being able to keep track of all the literature is a good regulariser to keep your brain operating at the level of abstraction that is likely to yield novel research insight.
And even then, some human communication (for physical coordination, with latency / throughput bottlenecks, across language barriers) takes place in few-shot without language in previously unseen situations.
Language is primarily a tool for communication rather than thought
“Language is a defining characteristic of our species, but the function, or functions, that it serves has been debated for centuries. Here we bring recent evidence from neuroscience and
@francoisfleuret
@ChrSzegedy
In the LLM space, there was nice work from INRIA recently on cultural evolution of stories among a population of large language models.
Very pleased to see
@CBSNews
describe with clarity and succinctness some of the societal-scale issues raised by myself and others in our recent GDM publication on the Ethics of AI Assistants:
@jeffclune
@ylecun
I have the same intuition as you
@jeffclune
: math is a (formal) language, and when I reason about this, most of what happens in my head is verbal ("use polar coordinates", "apply distance formulae" etc). If the problem were in 6D, reliance on language would be even more obvious.
This is interesting, perhaps LLMs do meta-learn some social learning capabilities. I wonder (a) to what extent these can be used beneficially in applications, and (b) whether one can create synthetic data which extends this effect to other desirable meta-cognitive capabilities.
Congrats to the whole team!
IIRC, our findings in this work blew my mind more than any other result.
This may be the first evidence for the existence of a *mechanism* by which sufficiently advanced AI systems would tend to become agentic (and thus have instrumental goals).
@polynoamial
Unless you can meta-learn a self-improvement operator from that data? In principle if we had a dataset for the practice of science, one could learn the principles of making new scientific discoveries (though one would still need real-world measurement to validate and iterate).
Brilliant to see further work on many-player zero-sum games. A few years ago my team at GDM also had a foray into this space, showing that 3-player zero-sum games often contain social dilemmas - that’s why multi-party negotiations are hard!
Ever wonder how to play multiplayer games (>2 players, such as Mahjong, Poker) well and what would be the ultimate solution? Check our paper on why classical equilibria and existing self-play systems are not enough, and how to address it:
Great leadership requires at least two things. (1) Humility of the leader to listen to ideas and course-correct when they are mistaken. (2) Generosity of the team to assume good intent and pro-actively provide feedback. Both are broken in contemporary UK politics.
Moravec's paradox is perhaps not so paradoxical when viewed through the lens of structured data. Mathematics, language, games provide abstractions which naturally compress and accumulate, accounting for the speed of cultural evolution (and the ability of large language models).
@soumithchintala
I keep talking about the Moravec paradox in my talks.
We have AI systems that can pass the bar exam, but where is the domestic robot that can clear up the dinner table and fill the dishwasher?
A task that any 10 year old can learn in one shot.
Obviously, we are missing something
This is an interesting, and perhaps not too surprising development. Thus far RLAIF has not benefitted fully from the kinds of ideas that led to open-ended self-improvement in the pre-LLM era (leagues, autocurricula, search). Once it does, the landscape will change.
A few big papers throwing question on "does RLAIF work" yesterday.
The first is a paper by
@archit_sharma97
is a pretty timely critique of RLAIF. It shows SFT on GPT 4 outputs > DPO + RLAIF on GPT4 ratings of GPT3.5 completions. A few things aren't surprising:
1. The most
Both AI scientists and politicans can learn a lot from this. In general, when you encounter multiple options you should first seek out an implement all possible Pareto improvements and *only then* seek trade-offs.
Excited to share our new paper on Pareto Optimal Preference Learning (POPL)! 🎉
POPL aims to better align AI with diverse human values by building diverse sets of reward functions or policies!
Work done with
@li_ding_
, Lee Spector and
@scottniekum
Large language model evaluations tend to focus on “correct answers”. Yet human communication is better modelled as providing responses which are “good enough” given the context and compute constraints. What will be the consequences of this mismatch?
@jparkerholder
Agreed. I’ve lost track of the time I’ve had to disclaim something. And like 100% of other people, I simply clicked agree and signed without reading it. What a waste of time for us and for the conference organisers.