AI could get really powerful soon and I worry we're underprepared. Funding research on AI capabilities+risks
@open_phil
(views my own), editor
@plannedobs
.
It's okay to disagree with us, but please don't ignore the women who have been publicly concerned about AI x-risk, like
@jesswhittles
,
@katjaGrace
and myself.
"A Taxonomy of AI Panic Facilitators"
A visualization of leading AI Doomers (X-risk open letters, media interviews & OpEds).
Some AI experts enable them, while others oppose them.
The gender dynamics are fucked up.
It says a lot about the panic itself.
Your thoughts?
Important article: The single most important data point that suggests "progress is unlikely to slow in the next 2-3y": GPT-4 cost ~$100M (probably less), and Alphabet has 1000x that much money in cash on hand:
Hi, I’m Ajeya! I work on reducing risks from misaligned AI,
@open_phil
. With AI being discussed more widely, I started a blog
@plannedobs
(I edit and sometimes write;
@KelseyTuoc
writes). I’m also planning to tweet more about AI issues.
This is very important to appreciate about the AI risk discourse IMO. I liked this chart from
@NateSilver538
's new book. A lot of people expect AI to top out somewhere between social media and the internet, well short of the rise of Homo sapiens.
Most AI “optimists” mostly seem skeptical that superintelligence is possible rather than genuinely optimistic that an unregulated rush to create it will end well.
Experts disagree wildly on how powerful LLMs could get in the near future. Very excited to announce two new
@open_phil
RFPs for projects that could shed light on this.
1st, a call for realistic+difficult benchmarks for LLM agents (e.g. OpenAI's "GPTs"):
6mo ago I did a mini DC tour asking policy wonks why they were skeptical of AI, many said stuff like “ChatGPT has no common sense, if you ask for walking directions to the moon it’ll answer instead of saying it’s impossible.” Often they were thinking of much weaker/older AIs.
It’s wild to me how detached from AI developments most normies are — a fellow parent told me yesterday that he didn’t think AI generation of high school essays is something we need to worry about within the span of our kids’ schooling.
I think future AI systems are likely to deserve rights/personhood, and it'd be very important to treat them well. But I think we're not ready to bring a new intelligent species into the world, both because we might mistreat them and because they might mistreat us.
Why do people fear AI? I hear three reasons:
1. Cynicism — the belief that it is rational not to cooperate
2. Humanism/racism — systematic bias against machines, denial of their potential moral worth and personhood
3. Conservatism — fear of change, fear of the other tribe
None
I agree with the letter of this but don't resonate with its implicit vibe. "Superhuman AI won't be built next year and won't just be an autoregressive LLM" != "It's far away and people sounding the alarm should chill out." 🧵
There is no question that AI will eventually reach and surpass human intelligence in all domains.
But it won't happen next year.
And it won't happen with the kind of Auto-Regressive LLMs currently in fashion (although they may constitute a component of it).
Strongly agree that there is no single "alignment problem" that you can "solve" once and for all. The goal is to keep avoiding catastrophic harm as we keep making more and powerful AI systems. This is not like solving P vs NP; it requires continuous engineering and policy effort.
One cannot just "solve the AI alignment problem."
Let alone do it in 4 years.
One doesn't just "solve" the safety problem for turbojets, cars, rockets, or human societies, either.
Engineering-for-reliability is always a process of continuous & iterative refinement.
A common criticism of people who are trying to stop existential risk from powerful future AI systems is that speculating about the future has poor feedback loops and doesn't work great.
It's unfortunate how discourse about dangerous capability evals often centers threats from today's models. Alice goes "Look, GPT-4 can hack stuff / scam people / make weapons," Bob goes "Nah, it's really bad at it." Bob's right! The ~entire worry is scaled-up future systems.
If you believe all four premises, *very intense* near-term risks look more plausible: dictators using AI to lock in control, states or terrorists using AI to develop bioweapons, AI companies accumulating unacceptable power, and AIs autonomously seizing control.
I think it's above the pay grade of scientists / companies to make the choice for everyone to let a more technologically-advanced civilization into our world. It could be good, but we should not be confident that it's good to hurtle toward this outcome as fast as possible.
The argument for fear of AI appears to be:
1. AI scientists are trying to make entities that are smarter than current people
2. If these entities are smarter than people, then they may become powerful
3. That would be really bad, something greatly to be feared, an “existential
3) LLMs are black boxes. Even when we literally have the systems in front of us, we don’t have great mechanistic stories about how and why they do what they do; we relate to them more like animals / aliens than normal software products.
The main driver of progress right now doesn’t operate on the timescale of “humans figuring out technical blockers to capabilities one by one,” it operates on the timescale of “humans making capital expenditures which bring on a whole host of capabilities at once."
Yes, I am ultimately most worried about systems that are goal-directed and able to take actions in the real world, and I think one important lever to reduce risk is to limit the extent to which powerful AI systems can pursue goals in the real world.
Look guys. If you're worried abt an extinction-level event from AI, one way to *create* that event is to enable LLMs to take actions in the real world. One way to *enable* LLMs to take actions is to give chatbots the ability to incorporate "plugins".
4) There's no clear power / impact ceiling. I think it's likely that this same mechanism of fast, unpredictable, black box progress could get all the way to systems that can make humans obsolete in all spheres (running companies and armies, doing science, manufacturing...)
Come work with me! Our team (just me and
@MaxNadeau__
right now) thinks through what research directions could put a big dent in risks from powerful AI systems and makes grants to research groups working on those directions.
I think this is true, but what's even more important is when GPT-2-sized models are as smart as GPT-4 is today, GPT-4-sized models will be *much smarter.* I think discussion of the "miniaturization trend" doesn't emphasize that enough
LLM model size competition is intensifying… backwards!
My bet is that we'll see models that "think" very well and reliably that are very very small. There is most likely a setting even of GPT-2 parameters for which most people will consider GPT-2 "smart". The reason current
We know what selection pressures they were formed under, but not what algorithms they implement. This makes it hard to get guarantees about behavior (and indeed you see tons of examples of weird behaviors in weird situations).
I appreciated the simplicity and straightforwardness of this statement, and strongly endorse it: risks as severe as human extinction are plausible outcomes of AI, we should think about that.
We just put out a statement:
“Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.”
Signatories include Hinton, Bengio, Altman, Hassabis, Song, etc.
🧵 (1/6)
1) Progress is unpredictable. We can’t tell what the next gen models will be capable of with the appropriate prompting and tools as in AutoGPT, LangChain, etc.
One thing that’s been bothering me a lot recently. There are these two notions floating around among people concerned about extreme AI risks that seem a little rough to reconcile 🧵
I agree with practically everything in this letter, especially the emphasis on the speed of progress, the competitive pressure to defer to increasingly capable AI systems, and the need for regulation (not just safety techniques):
2) Progress is fast and could get even faster. All this unpredictable progress has mainly been driven by making training runs and models bigger. We are in the middle of a spend overhang where it’s very easy to keep doing that for another 2 OOMs, maybe more.
Before the stage where an AI could trivially invent microscopic self-replicating factories, I'd guess AIs could collectively secure control over critical infrastructure that looks much more like today's (mines, fabs, factories, robots).
An artificial superintelligence will not wipe out humanity until It has built Its own self-replicating factories; It won't destroy Its own infrastructure without a replacement.
As bacteria demonstrate, self-replicating factories can be 5 microns long by 2 microns diameter.
(This process is not instantaneous, because it’s bottlenecked on like hardware procurement and cluster setup and other not-instantaneous processes, but those are like 1-3 year timescale endeavors, whereas we're used to thinking of ambitious R&D taking decades.)
We have some benchmarks (like standardized tests), but it's confusing to translate benchmark perf into real world impacts, since they often make weird assumptions like “the model gets one shot to do this and can’t go back and edit its work or use Wolfram Alpha.”
The speed makes this unpredictability way scarier. A bunch of unpredictable capabilities could come online in just a couple of years, and we might not be able to react and adapt on that kind of timescale.
I'm really excited to see measurements of LLM agents on real-world tasks; it allows us to make progress on key disagreements. I'll lay down a forecast: >50% chance that >50% of the tasks will be solved by EOY 2024 (current accuracy is ~10%).
🤖There have been recent exciting demos of agents that navigate the web and perform tasks for us. But how well do they work in practice?
🔊To answer this, we built WebArena, a realistic and reproducible web environment with 4+ real-world web apps for benchmarking useful agents🧵
Biorisk is really important and badly neglected. I worry though that within a couple decades companies could develop the kind of AI system that could act on its *own* to invent / deploy bioweapons (among other things). At that point, most biorisk would be downstream of AI IMO.
The AI Doomer argument that AI development should be paused, prevented, or airstrike'd to save humanity misses the much more likely doom vector: Bioweapons.
Perhaps superintelligent AI could do harm, but a couple bad dudes with commoditized medtech tools are so much scarier.
I have not listened to this, but I feel I have the right to RT because I've also spent 7+ hours having my worldview changed by talking to Carl. Both parts of this very-likely-fascinating-and-thorough interview are now out!
No part of my worldview is the same after talking with Carl Shulman
Maybe 3 people in the world have thought as rigorously about so many interesting topics
Part 1 is about Carl's model of an intelligence explosion
(We ended up talking for 8 hrs, so I'm splitting this episode
This last premise is the most controversial premise; I wind up disagreeing the most with people about this. I think it is genuinely the most shaky premise, but it's what I'd bet on.
I'm a big fan of efforts to forecast AI capabilities. Here's a good 2 year update on a forecasting competition run by
@JacobSteinhardt
to forecast MATH and MMLU: I also recommend his earlier one year in update!
Of course some possibilities are too remote to worry about, but powerful AI seems likely soon. Companies are actively trying to make AI systems as capable as possible and they seem set to succeed. (1/2)
We cannot rule out the possibility that radio communication will alert hostile alien superintelligences to our whereabouts, triggering an invasion posing an existential risk to humanity. Therefore we must carefully regulate radio technology until we have better planetary defenses
New post on Planned Obsolescence, written with
@KelseyTuoc
: Experts were surprised by progress in LLMs, and I think there's probably more surprise coming: for one thing, most people don't seem to be pricing in another GPT3 -> GPT4 size scaleup.
Technical researchers often don't have govt careers on their radar, but govts need technical expertise to properly regulate complex and fast-moving tech. UK AISI is an unusual opportunity to do very technical work that plugs directly into govt decisionmaking!
The UK's AI Safety Institute is hiring. In my (biased) view, this one of the best places to do AI research/engineering for the public good, with top talent & the resources / backing / access of gov. Super excited for even more technical ppl to join gov :) 🧵 on open roles! 1/9
In this post,
@KelseyTuoc
and I introduce a distinction we've found helpful for thinking about LLM progress: you can make LLMs more useful both by *scaling* them up, and by doing *schlep* to turn the "raw models" into full-blown AI *systems*:
Appreciated this exhaustive representative list of how one programmer and ML researchers uses LLMs: Surprisingly little of this kind of content! We're funding some in-depth power-user interviews to create more 🤞
But I'm not aware of anyone who successfully took complex deliberate *non-obvious* action many years or decades ahead of time on the basis of speculation about how some technology would change the future.
A few months ago, I recorded an interview with Freakonomics for a sequence of AI-themed episodes they're doing: I had a lot of fun, and I think the episode turned out great! Thanks to Adam, Julie, and the team!
Like, sometimes people are surprisingly right about the general contours of the tech we might develop (cf ) and I think sometimes the failure rate of this kind of forecasting is exaggerated by critics.
Excited to share a new blog post on Planned Obsolescence (first one in a while!) by my colleague
@lucafrighetti
! Luca says dangerous capability tests need to get *way* harder:
@open_phil
And 2nd, a call for a wide variety of research projects that could help shed light on what kind of real-world impact LLM systems could have:
Appreciated this pushback on the Nature editorial from a skeptic. I take "AI doom" very seriously but am very confused about the nature/extent of it. Hopefully people with a range of views on the question can talk about it and collectively figure it out.
I am skeptical of AI doom talk but I thought this was not a good editorial
First there is something off about a scientific journal saying “stop talking” about a potential risk
Really they shouldn’t be telling us to stop talking about anything
And....yeah, sincerely trying to figure out in some detail how future AI developments might play out and impact society is a maddening, confusing activity that feels like trying to wring water from rocks.
At least, not something that would outperform some strategy like "try to generally learn about the technology and be close to the epicenter of its development, and be watchful and flexible and look for ways to help."
METR put out a standard way of defining tasks to evaluate agents that can interact with the world over a computer (e.g. AutoGPT etc): Currently being used by METR + UK AI safety institute + other orgs. Applicants to our RFP might find it useful!
Experts disagree wildly on how powerful LLMs could get in the near future. Very excited to announce two new
@open_phil
RFPs for projects that could shed light on this.
1st, a call for realistic+difficult benchmarks for LLM agents (e.g. OpenAI's "GPTs"):
Strongly agreed on "To scientifically evaluate claims of humanlike and even superhuman machine intelligence, we need...better experimental methods and benchmarks." I really wish we were measuring suits of tasks more tightly correlated with real-world impacts.
2) I think we have reason to think (based on this kind of thoughtful speculation!) that *AI might go from pretty-powerful to super-powerful too quickly to react to in real time.*
If you have experience communicating thoughtfully, honestly, and compellingly about important topics, apply for the exec-level Comms Director role at
@open_phil
, even if you don't have a traditional PR background!
@jeremyphoward
Agree, but historically we've been able to name specific massive capabilities gaps that could make us confident NNs were far from major real-world impacts (e.g. "they can't say coherent sentences," "they can't do math / logic"). It's a lot less clear now IMO.
Handy graphic from the blog post version of this paper on arguments for and against "explosive economic growth" from AI (aka at least tenfold increase in growth rates aka >= 30% YoY growth).
We assess if AI will accelerate economic growth by as much as growth accelerated during the industrial revolution, digging into growth theory, bottlenecks, feasibility of regulation, AI reliability/alignment, etc. Takeaway: acceleration looks plausible
Regulations on AI are currently very light compared to e.g. the aviation industry and I think the chance of catastrophic harm in the next 10y is >10x higher than an airplane crash (and our whole civilization would be on board).
If we make it through a transition to powerful AI, *most* of the value-added will probably come from people who are doing concrete, grounded things (ML research grounded in near-term problems, policy development grounded in near-term concerns, etc).
But I do it anyway because: 1) the stakes of potentially developing AI that can obsolete humans are high enough that I think *some* people in the world should be doing this activity even if it's in some sense low-ROI;
It's great to see this new cyber benchmark from
@percyliang
and co that we funded through our RFP! Funding ML is unfairly gratifying — we barely manage to get out the post announcing the grant by the time the grantee has a whole paper out 😂
The EU AI Office is looking for EU citizens with technical backgrounds to help implement the newly passed EU AI Act effectively. If you're interested, apply by Mar 27th!
Interested in joining the 🇪🇺 AI Office as technical staff?
Tune in to this online event to learn more about the work of the
#AI
Office, the latest job opportunities & get your questions answered.
🗓️ 20 March
⏰ 18:00 CET
Register now 👇
@mattyglesias
Hmm, I think there’s enormous upside from automating all the good things humans do to make everyone wealthier and healthier and more empowered;
@KelseyTuoc
wrote about that here:
I do think it's super unclear how far we have to go. Performance on a bunch of static benchmarks is high but it's really unclear to me how to translate that into real world impact; it seems *plausible* it's ~10y or so though, and the massive uncertainty scares me.
This is why I disagree with the "sparks of AGI" framing. There is no more reason to consider the current moment an inflection point than any previous moment in the history of ML (or technology). And we don't know how far we have left to go.
Loved this post () from
@mattyglesias
but IMO the nukes analogy misses how all-encompassing human-level AI would be: more like "industrial capacity" or "human capital" — important for war, but also everything else (and makes openness seem less weird).
We've rec'd $20M to agent benchmarks so far! 🎉 We're really excited to see what all our grantees build, and we're winding down the RFP to focus on new projects. New EOIs (300 words max) must be submitted *by Jul 26 at 11:59PM Eastern*:
Two updates on these funding opportunities:
1) We replaced our longer applications with a 300 word EOI 🎉
2) The broader RFP on studying+forecasting LLM impacts will be on hiatus after May 3 (the agent benchmarks RFP is still going strong!)
New Planned Obsolescence post from my colleague Tom Davidson!
In this post, Tom explains why most individual technologies don't accelerate economic growth, and why AI could be different.
Two updates on these funding opportunities:
1) We replaced our longer applications with a 300 word EOI 🎉
2) The broader RFP on studying+forecasting LLM impacts will be on hiatus after May 3 (the agent benchmarks RFP is still going strong!)
Experts disagree wildly on how powerful LLMs could get in the near future. Very excited to announce two new
@open_phil
RFPs for projects that could shed light on this.
1st, a call for realistic+difficult benchmarks for LLM agents (e.g. OpenAI's "GPTs"):
I think ultimately there will be a complicated and “boring” reconciliation here that involves being more open about architectures and training strategies, doing some stuff but not all stuff in secret, still trying to secure model weights, eating some costs on both sides, etc.
But IMO the prima facie case for AI risk makes reference to training choices. If we had a robust+mature science that showed that even OOD and tail behaviors are independent of training processes, we could ignore this. But we can't do much of that science outside companies.
I think a compelling safety case will often involve making statements about the training procedure and architecture. For example, it could matter whether models have persistent long-term memory, whether they’re trained with RL on metrics like medium-run profit, etc.
I’m honoured to be appointed as the Chair of the UK's AI Foundation Model Taskforce.
A thread on why I'm doing this and how you might be able to help us.
We've already gotten several exciting applications to these RFPs, and we want many more! We'll be hosting a webinar Wed at 10 am Pacific (in 2 days!) to help potential applicants better understand what we're looking for and how our process works:
Experts disagree wildly on how powerful LLMs could get in the near future. Very excited to announce two new
@open_phil
RFPs for projects that could shed light on this.
1st, a call for realistic+difficult benchmarks for LLM agents (e.g. OpenAI's "GPTs"):
Agree AI safety is highly interdsiciplinary. (I'd add infosecurity to this list.) A lot of work goes beyond academic disciplines too. E.g. for security it's not academic study of security that matters most; it's security practices in AI labs.
Personally I have a >2.5% chance by 2029, something more like 15%? A lot of the disagreement here might be that I think it's unlikely we'll stay at "roughly human level" for very long, bc human-level AI can accelerate research into superhuman AI:
@AISafetyMemes
Very wide probability bars honestly. 95% confidence on timelines 2029-2200 for ASI, (I have significant probability mass on AGI staying roughly-human-level for a while)
p(doom) around 0.1? Which of course makes it quite important to take AI risk seriously.
I would dearly love to not be in a world where I'm trying incredibly hard to finely interpret all this weak and ambiguous evidence we have about how AI might unfold.
This week on Hard Fork:
• Digging into the Surgeon General’s new warning about social media and adolescents
• A conversation with the great
@ajeya_cotra
about AI’s existential risks
• HatGPT returns!
But I don’t know exactly what the reconciliation would be on the substance, and I suspect a reconciliation that’s right on the substance might end up more pro openness than most companies would be willing to go for.
SOTA systems like ChatGPT are already more than just autoregressive LLMs. They're trained with RL on top of autoregressive (i.e. predict-the-next-token) loss, and the customer-facing product has all sorts of bells and whistles (often hand-engineered) on top of the trained model.
Great paper and post — focused on agent developers and application developers, but benchmarks that actually measure what they mean to and make it hard to cheat are just as indispensable for policymakers and safety researchers
Blog post with additional context about our new paper "AI Agents That Matter":
- What does the term agent mean? Is it just a buzzword?
- Do agents even work?
- How does the poor reproducibility of agent evaluations compare with other areas of ML?
Staying close, paying attention, watching, and being flexible probably works better if we have the option. But crucial developments may happen so quickly that we don't have time to absorb them.
I think externally-evaluated safety cases are super important, and to me it feels in tension with placing a lot of emphasis on avoiding sharing information about how SOTA systems are trained+built.
I don't think you need to be particularly confident about what concrete technologies become possible with far-superhuman AI in order to buy a story where humans lose control to AI.
I still think *most* people worried about AI risk shouldn't spend a lot of time engaging in detailed speculation about how AI developments might unfold.
In five years, I think a significant percentage of current MTurk tasks will probably be done by AI (would have to look at the tasks to get to a number). But I think that's short of automating all the most impactful things humans do (e.g. R&D, corporate decisionmaking, etc).
This can be measured precisely: five years from now, what percentage of Mechanical Turk problems are being solved by AI?
If your AI can do 100% of the solvable problems, it means you’ve achieved superintelligence, and you stand to earn all the money.
My bet is, nowhere close.
Horizon is a great opportunity to get a foot in the door in AI policy. I especially hope ML PhD students and other technical folks consider it! Applications close Aug 30:
Apply to become a Horizon Fellow! Applications close Aug. 30.
Curious what it is like to work on AI and biotech policy in DC?
The Horizon Fellowship is a paid opportunity to work in a think tank, Congress or executive agency. Learn more here:
Notion 1: It's risky to widely disseminate information about architectures and training strategies for cutting-edge AI systems, because that advances AI R&D and means uncontrollably powerful AI is developed sooner with less time to prepare with safety+governance measures.
Really impressive! I know it's a temporary artifact that will be ironed out soon, but I actually like the wobbly trippy quality of current AI-generated videos. I'll be curious to see how this impacts the film industry.
AI just changed the game in filmmaking 📽️
Creators can now generate cinema quality short films, trailers, and teasers with 100% AI in just a few hours 🤯
10 favorite examples that will blow your mind:
Notion 2: AI companies should have to make "safety cases" arguing that their SOTA systems are safe enough to train and deploy, and neutral third party experts should evaluate whether those safety cases pass muster.
I expect there to be lots of gains from synthetic data in domains where the model can come up with problems for itself where the solution is checkable but still difficult to find, e.g. stating theorems and attempting to prove them or making up programming challenges.
A popular view. I get it — learning w/o human knowledge seems implausible, even insulting. But let's wait and see.
We already know this is false in some domains like game playing — synthetic data *alone* (self play) yields superhuman agents. In which other domains can it help?
This is a great opportunity, I especially encourage you to apply if you have a technical background and want to apply your skills to help shape AI policy!
Applications for the 2024 Horizon Fellowship are now open (deadline Sept 15th)!
Do you have a background relevant to emerging tech policy? Interested in up to 2 years of funding to do impactful work on AI or biosecurity in DC? — Learn more here:
Informal adversarial collaboration testing GPT-4 with access to plug-ins (e.g. Wolfram Alpha) on math problems: Highlights that there's low-hanging fruit in training models to better use plug-ins