Ajeya Cotra @ajeya_cotra Twitter profile

Last Seen Profiles

@cukienaknikmati

@stwmaniax

@Hang_in_O

@raptors_nj

@galadancer

@_theurbanafro_

@jandakembangstw

@SWCulture

@_chismosa_

@yawdalayaw

@kimikkk2

@telescoper

@TrialNz

@Sunny_Singh10

@DelCisnech

@SUzumcu

@mohaMd6__

@RaceTraitor_

@soapy_fate

@kai2187

@Juan1437461

@HidaMughal

@phu_25025635

@Arabian_Analyst

@SteveStoute

@YeetD0ntJeet

@Overwatch_data

@amar_mahbubani

@MiriYeminiGroup

@mimayfair

@BinorRaja

@shaunabing

@UydXxxx9734

@SS59867511

@ApparelSpire

@lizwathuti

Ajeya Cotra

@ajeya_cotra

1 year

It's okay to disagree with us, but please don't ignore the women who have been publicly concerned about AI x-risk, like @jesswhittles , @katjaGrace and myself.

Nirit Weiss-Blatt, PhD

@DrTechlash

1 year

"A Taxonomy of AI Panic Facilitators" A visualization of leading AI Doomers (X-risk open letters, media interviews & OpEds). Some AI experts enable them, while others oppose them. The gender dynamics are fucked up. It says a lot about the panic itself. Your thoughts?

122

192

755

18

56

532

Ajeya Cotra

@ajeya_cotra

1 year

Four reasons recent LLM progress makes me think extreme risks could emerge soon 🧵

16

94

474

Ajeya Cotra

@ajeya_cotra

1 year

Important article: The single most important data point that suggests "progress is unlikely to slow in the next 2-3y": GPT-4 cost ~$100M (probably less), and Alphabet has 1000x that much money in cash on hand:

4 Charts That Show Why AI Progress Is Unlikely to Slow Down

Three major inputs are the key to understanding how AI will likely progress

time.com

27

84

458

Ajeya Cotra

@ajeya_cotra

1 year

Hi, I’m Ajeya! I work on reducing risks from misaligned AI, @open_phil . With AI being discussed more widely, I started a blog @plannedobs (I edit and sometimes write; @KelseyTuoc writes). I’m also planning to tweet more about AI issues.

About

This is a blog about AI futurism and AI alignment. It's edited by Ajeya Cotra, and written primarily by her and Kelsey Piper (although we may solicit guest posts in the future). Our first post has...

www.planned-obsolescence.org

18

52

332

Ajeya Cotra

@ajeya_cotra

5 days

This is very important to appreciate about the AI risk discourse IMO. I liked this chart from @NateSilver538 's new book. A lot of people expect AI to top out somewhere between social media and the internet, well short of the rise of Homo sapiens.

Matthew Yglesias

@mattyglesias

5 days

Most AI “optimists” mostly seem skeptical that superintelligence is possible rather than genuinely optimistic that an unregulated rush to create it will end well.

15

19

205

30

47

321

Ajeya Cotra

@ajeya_cotra

1 year

I really liked this blog post by distinguished ML researcher Yoshua Bengio, among the clearest explanations of AI existential risk I've encountered:

How Rogue AIs may Arise - Yoshua Bengio

yoshuabengio.org

5

63

245

Ajeya Cotra

@ajeya_cotra

10 months

Experts disagree wildly on how powerful LLMs could get in the near future. Very excited to announce two new @open_phil RFPs for projects that could shed light on this. 1st, a call for realistic+difficult benchmarks for LLM agents (e.g. OpenAI's "GPTs"):

[On hiatus] Request for proposals: benchmarking LLM agents on consequential real-world tasks | Open...

Update, 7/26/24: This RFP is now on hiatus; we are no longer accepting new expressions of interest. Update, 7/12/24: We will put this RFP on hiatus starting on 7/26/24 in order to focus on other...

www.openphilanthropy.org

12

60

229

Ajeya Cotra

@ajeya_cotra

3 months

6mo ago I did a mini DC tour asking policy wonks why they were skeptical of AI, many said stuff like “ChatGPT has no common sense, if you ask for walking directions to the moon it’ll answer instead of saying it’s impossible.” Often they were thinking of much weaker/older AIs.

Matthew Yglesias

@mattyglesias

3 months

It’s wild to me how detached from AI developments most normies are — a fellow parent told me yesterday that he didn’t think AI generation of high school essays is something we need to worry about within the span of our kids’ schooling.

64

51

1K

14

18

213

Ajeya Cotra

@ajeya_cotra

1 year

I think future AI systems are likely to deserve rights/personhood, and it'd be very important to treat them well. But I think we're not ready to bring a new intelligent species into the world, both because we might mistreat them and because they might mistreat us.

Richard Sutton

@RichardSSutton

1 year

Why do people fear AI? I hear three reasons: 1. Cynicism — the belief that it is rational not to cooperate 2. Humanism/racism — systematic bias against machines, denial of their potential moral worth and personhood 3. Conservatism — fear of change, fear of the other tribe None

83

27

130

35

11

214

Ajeya Cotra

@ajeya_cotra

4 months

I agree with the letter of this but don't resonate with its implicit vibe. "Superhuman AI won't be built next year and won't just be an autoregressive LLM" != "It's far away and people sounding the alarm should chill out." 🧵

Yann LeCun

@ylecun

4 months

There is no question that AI will eventually reach and surpass human intelligence in all domains. But it won't happen next year. And it won't happen with the kind of Auto-Regressive LLMs currently in fashion (although they may constitute a component of it).

227

324

3K

3

30

201

Ajeya Cotra

@ajeya_cotra

1 year

Strongly agree that there is no single "alignment problem" that you can "solve" once and for all. The goal is to keep avoiding catastrophic harm as we keep making more and powerful AI systems. This is not like solving P vs NP; it requires continuous engineering and policy effort.

Yann LeCun

@ylecun

1 year

One cannot just "solve the AI alignment problem." Let alone do it in 4 years. One doesn't just "solve" the safety problem for turbojets, cars, rockets, or human societies, either. Engineering-for-reliability is always a process of continuous & iterative refinement.

157

313

3K

6

24

189

Ajeya Cotra

@ajeya_cotra

1 year

A common criticism of people who are trying to stop existential risk from powerful future AI systems is that speculating about the future has poor feedback loops and doesn't work great.

10

32

177

Ajeya Cotra

@ajeya_cotra

4 months

It's unfortunate how discourse about dangerous capability evals often centers threats from today's models. Alice goes "Look, GPT-4 can hack stuff / scam people / make weapons," Bob goes "Nah, it's really bad at it." Bob's right! The ~entire worry is scaled-up future systems.

6

17

148

Ajeya Cotra

@ajeya_cotra

1 year

If you believe all four premises, *very intense* near-term risks look more plausible: dictators using AI to lock in control, states or terrorists using AI to develop bioweapons, AI companies accumulating unacceptable power, and AIs autonomously seizing control.

10

7

140

Ajeya Cotra

@ajeya_cotra

1 year

I think it's above the pay grade of scientists / companies to make the choice for everyone to let a more technologically-advanced civilization into our world. It could be good, but we should not be confident that it's good to hurtle toward this outcome as fast as possible.

Richard Sutton

@RichardSSutton

1 year

The argument for fear of AI appears to be: 1. AI scientists are trying to make entities that are smarter than current people 2. If these entities are smarter than people, then they may become powerful 3. That would be really bad, something greatly to be feared, an “existential

131

9

125

11

7

141

Ajeya Cotra

@ajeya_cotra

1 year

3) LLMs are black boxes. Even when we literally have the systems in front of us, we don’t have great mechanistic stories about how and why they do what they do; we relate to them more like animals / aliens than normal software products.

3

9

130

Ajeya Cotra

@ajeya_cotra

1 year

The main driver of progress right now doesn’t operate on the timescale of “humans figuring out technical blockers to capabilities one by one,” it operates on the timescale of “humans making capital expenditures which bring on a whole host of capabilities at once."

1

8

127

Ajeya Cotra

@ajeya_cotra

1 year

Yes, I am ultimately most worried about systems that are goal-directed and able to take actions in the real world, and I think one important lever to reduce risk is to limit the extent to which powerful AI systems can pursue goals in the real world.

MMitchell

@mmitchell_ai

1 year

Look guys. If you're worried abt an extinction-level event from AI, one way to *create* that event is to enable LLMs to take actions in the real world. One way to *enable* LLMs to take actions is to give chatbots the ability to incorporate "plugins".

37

105

589

4

12

116

Ajeya Cotra

@ajeya_cotra

1 year

Do you know what GPT-4 can do? A game from Nicholas Carlini:

7

18

116

Ajeya Cotra

@ajeya_cotra

1 year

4) There's no clear power / impact ceiling. I think it's likely that this same mechanism of fast, unpredictable, black box progress could get all the way to systems that can make humans obsolete in all spheres (running companies and armies, doing science, manufacturing...)

8

5

114

Ajeya Cotra

@ajeya_cotra

10 months

Come work with me! Our team (just me and @MaxNadeau__ right now) thinks through what research directions could put a big dent in risks from powerful AI systems and makes grants to research groups working on those directions.

New roles on my team: come build Open Phil's technical AI safety program with me! — EA Forum

Open Phil announced two weeks ago that we’re hiring for over 20 roles across our teams working on global catastrophic risk reduction — and we’ll answ…

forum.effectivealtruism.org

5

19

112

Ajeya Cotra

@ajeya_cotra

1 month

I think this is true, but what's even more important is when GPT-2-sized models are as smart as GPT-4 is today, GPT-4-sized models will be *much smarter.* I think discussion of the "miniaturization trend" doesn't emphasize that enough

Andrej Karpathy

@karpathy

1 month

LLM model size competition is intensifying… backwards! My bet is that we'll see models that "think" very well and reliably that are very very small. There is most likely a setting even of GPT-2 parameters for which most people will consider GPT-2 "smart". The reason current

201

960

8K

5

14

113

Ajeya Cotra

@ajeya_cotra

1 year

We know what selection pressures they were formed under, but not what algorithms they implement. This makes it hard to get guarantees about behavior (and indeed you see tons of examples of weird behaviors in weird situations).

1

2

104

Ajeya Cotra

@ajeya_cotra

1 year

I appreciated the simplicity and straightforwardness of this statement, and strongly endorse it: risks as severe as human extinction are plausible outcomes of AI, we should think about that.

Dan Hendrycks

@DanHendrycks

1 year

We just put out a statement: “Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.” Signatories include Hinton, Bengio, Altman, Hassabis, Song, etc. 🧵 (1/6)

116

377

1K

2

4

105

Ajeya Cotra

@ajeya_cotra

1 year

1) Progress is unpredictable. We can’t tell what the next gen models will be capable of with the appropriate prompting and tools as in AutoGPT, LangChain, etc.

2

4

98

Ajeya Cotra

@ajeya_cotra

5 months

One thing that’s been bothering me a lot recently. There are these two notions floating around among people concerned about extreme AI risks that seem a little rough to reconcile 🧵

8

12

98

Ajeya Cotra

@ajeya_cotra

10 months

I agree with practically everything in this letter, especially the emphasis on the speed of progress, the competitive pressure to defer to increasingly capable AI systems, and the need for regulation (not just safety techniques):

2

6

95

Ajeya Cotra

@ajeya_cotra

1 year

2) Progress is fast and could get even faster. All this unpredictable progress has mainly been driven by making training runs and models bigger. We are in the middle of a spend overhang where it’s very easy to keep doing that for another 2 OOMs, maybe more.

2

94

Ajeya Cotra

@ajeya_cotra

1 year

Before the stage where an AI could trivially invent microscopic self-replicating factories, I'd guess AIs could collectively secure control over critical infrastructure that looks much more like today's (mines, fabs, factories, robots).

Eliezer Yudkowsky ⏹️

@ESYudkowsky

1 year

An artificial superintelligence will not wipe out humanity until It has built Its own self-replicating factories; It won't destroy Its own infrastructure without a replacement. As bacteria demonstrate, self-replicating factories can be 5 microns long by 2 microns diameter.

111

39

561

6

5

91

Ajeya Cotra

@ajeya_cotra

1 year

(This process is not instantaneous, because it’s bottlenecked on like hardware procurement and cluster setup and other not-instantaneous processes, but those are like 1-3 year timescale endeavors, whereas we're used to thinking of ambitious R&D taking decades.)

1

3

88

Ajeya Cotra

@ajeya_cotra

1 year

We have some benchmarks (like standardized tests), but it's confusing to translate benchmark perf into real world impacts, since they often make weird assumptions like “the model gets one shot to do this and can’t go back and edit its work or use Wolfram Alpha.”

3

4

87

Ajeya Cotra

@ajeya_cotra

1 year

The speed makes this unpredictability way scarier. A bunch of unpredictable capabilities could come online in just a couple of years, and we might not be able to react and adapt on that kind of timescale.

2

1

87

Ajeya Cotra

@ajeya_cotra

1 year

I thought this profile of @DanHendrycks by @dscharfGlobe was great and moving:

Dan Hendrycks wants to save us from an AI catastrophe. He’s not sure he’ll succeed. - The Boston...

An evangelical turned computer scientist has articulated how technology could all go wrong. Now he needs to figure out how to make it right.

www.bostonglobe.com

0

10

81

Ajeya Cotra

@ajeya_cotra

1 year

Really enjoyed taping this interview, and I think it's a good window into a lot of the things on my mind!

2

7

81

Ajeya Cotra

@ajeya_cotra

1 year

I'm really excited to see measurements of LLM agents on real-world tasks; it allows us to make progress on key disagreements. I'll lay down a forecast: >50% chance that >50% of the tasks will be solved by EOY 2024 (current accuracy is ~10%).

Shuyan Zhou

@shuyanzhxyc

1 year

🤖There have been recent exciting demos of agents that navigate the web and perform tasks for us. But how well do they work in practice? 🔊To answer this, we built WebArena, a realistic and reproducible web environment with 4+ real-world web apps for benchmarking useful agents🧵

8

72

377

3

6

81

Ajeya Cotra

@ajeya_cotra

1 year

Biorisk is really important and badly neglected. I worry though that within a couple decades companies could develop the kind of AI system that could act on its *own* to invent / deploy bioweapons (among other things). At that point, most biorisk would be downstream of AI IMO.

Palmer Luckey

@PalmerLuckey

1 year

The AI Doomer argument that AI development should be paused, prevented, or airstrike'd to save humanity misses the much more likely doom vector: Bioweapons. Perhaps superintelligent AI could do harm, but a couple bad dudes with commoditized medtech tools are so much scarier.

106

113

1K

8

4

79

Ajeya Cotra

@ajeya_cotra

1 year

I have not listened to this, but I feel I have the right to RT because I've also spent 7+ hours having my worldview changed by talking to Carl. Both parts of this very-likely-fascinating-and-thorough interview are now out!

Dwarkesh Patel

@dwarkesh_sp

1 year

No part of my worldview is the same after talking with Carl Shulman Maybe 3 people in the world have thought as rigorously about so many interesting topics Part 1 is about Carl's model of an intelligence explosion (We ended up talking for 8 hrs, so I'm splitting this episode

29

44

491

1

4

74

Ajeya Cotra

@ajeya_cotra

1 year

This last premise is the most controversial premise; I wind up disagreeing the most with people about this. I think it is genuinely the most shaky premise, but it's what I'd bet on.

4

2

74

Ajeya Cotra

@ajeya_cotra

1 year

I'm a big fan of efforts to forecast AI capabilities. Here's a good 2 year update on a forecasting competition run by @JacobSteinhardt to forecast MATH and MMLU: I also recommend his earlier one year in update!

AI Forecasting: Two Years In

Two years ago, I commissioned forecasts for state-of-the-art performance on several popular ML benchmarks. Forecasters were asked to predict state-of-the-art performance on June 30th of 2022, 2023,...

bounded-regret.ghost.io

1

11

72

Ajeya Cotra

@ajeya_cotra

1 year

Of course some possibilities are too remote to worry about, but powerful AI seems likely soon. Companies are actively trying to make AI systems as capable as possible and they seem set to succeed. (1/2)

David Pfau

@pfau

1 year

We cannot rule out the possibility that radio communication will alert hostile alien superintelligences to our whereabouts, triggering an invasion posing an existential risk to humanity. Therefore we must carefully regulate radio technology until we have better planetary defenses

113

223

2K

1

4

62

Ajeya Cotra

@ajeya_cotra

1 year

New post on Planned Obsolescence, written with @KelseyTuoc : Experts were surprised by progress in LLMs, and I think there's probably more surprise coming: for one thing, most people don't seem to be pricing in another GPT3 -> GPT4 size scaleup.

Language models surprised us

Most experts were surprised by progress in language models in 2022 and 2023. There may be more surprises ahead, so experts should register their forecasts now about 2024 and 2025.

www.planned-obsolescence.org

5

17

61

Ajeya Cotra

@ajeya_cotra

1 year

Excellent post by @JacobSteinhardt trying to forecast the abilities of models that could be trained in 2030:

What will GPT-2030 look like?

GPT-4 surprised many people with its abilities at coding, creative brainstorming, letter-writing, and other skills. How can we be less surprised by developments in machine learning? In this post,...

bounded-regret.ghost.io

1

8

55

Ajeya Cotra

@ajeya_cotra

1 year

I was recently featured in a pair of articles in The Guardian about how AI could massively improve the world or destory it.

1

4

56

Ajeya Cotra

@ajeya_cotra

8 months

Technical researchers often don't have govt careers on their radar, but govts need technical expertise to properly regulate complex and fast-moving tech. UK AISI is an unusual opportunity to do very technical work that plugs directly into govt decisionmaking!

Xander Davies

@alxndrdavies

8 months

The UK's AI Safety Institute is hiring. In my (biased) view, this one of the best places to do AI research/engineering for the public good, with top talent & the resources / backing / access of gov. Super excited for even more technical ppl to join gov :) 🧵 on open roles! 1/9

9

83

229

2

13

54

Ajeya Cotra

@ajeya_cotra

11 months

In this post, @KelseyTuoc and I introduce a distinction we've found helpful for thinking about LLM progress: you can make LLMs more useful both by *scaling* them up, and by doing *schlep* to turn the "raw models" into full-blown AI *systems*:

Scale, schlep, and systems

This startlingly fast progress in LLMs was driven both by scaling up LLMs and doing schlep to make usable systems out of them. We think scale and schlep will both improve rapidly.

www.planned-obsolescence.org

2

19

53

Ajeya Cotra

@ajeya_cotra

20 days

Appreciated this exhaustive representative list of how one programmer and ML researchers uses LLMs: Surprisingly little of this kind of content! We're funding some in-depth power-user interviews to create more 🤞

0

2

51

Ajeya Cotra

@ajeya_cotra

1 year

But I'm not aware of anyone who successfully took complex deliberate *non-obvious* action many years or decades ahead of time on the basis of speculation about how some technology would change the future.

12

3

50

Ajeya Cotra

@ajeya_cotra

1 year

A few months ago, I recorded an interview with Freakonomics for a sequence of AI-themed episodes they're doing: I had a lot of fun, and I think the episode turned out great! Thanks to Adam, Julie, and the team!

New Technologies Always Scare Us. Is A.I. Any Different? - Freakonomics

freakonomics.com

3

2

50

Ajeya Cotra

@ajeya_cotra

1 year

Like, sometimes people are surprisingly right about the general contours of the tech we might develop (cf ) and I think sometimes the failure rate of this kind of forecasting is exaggerated by critics.

The Track Record of Futurists Seems ... Fine

We scored mid-20th-century sci-fi writers on nonfiction predictions. They weren't great, but weren't terrible either. Maybe doing futurism works fine.

www.cold-takes.com

2

49

Ajeya Cotra

@ajeya_cotra

7 days

Excited to share a new blog post on Planned Obsolescence (first one in a while!) by my colleague @lucafrighetti ! Luca says dangerous capability tests need to get *way* harder:

Dangerous capability tests should be harder

We should spend less time proving that today’s AIs are safe and more time figuring out how to tell if tomorrow’s AIs are dangerous.

www.planned-obsolescence.org

3

11

47

Ajeya Cotra

@ajeya_cotra

10 months

@open_phil And 2nd, a call for a wide variety of research projects that could help shed light on what kind of real-world impact LLM systems could have:

[On hiatus] RFP on studying and forecasting the real-world impacts of systems built from LLMs |...

Update, 5/3/24: This RFP is now on hiatus. Update, 4/5/24: We have replaced the request for a proposal PDF with a much shorter expression of interest (EOI) form [no longer active]. Submissions should...

www.openphilanthropy.org

3

1

47

Ajeya Cotra

@ajeya_cotra

1 year

Appreciated this pushback on the Nature editorial from a skeptic. I take "AI doom" very seriously but am very confused about the nature/extent of it. Hopefully people with a range of views on the question can talk about it and collectively figure it out.

Itai Sher

@itaisher

1 year

I am skeptical of AI doom talk but I thought this was not a good editorial First there is something off about a scientific journal saying “stop talking” about a potential risk Really they shouldn’t be telling us to stop talking about anything

8

17

179

0

5

47

Ajeya Cotra

@ajeya_cotra

1 year

And....yeah, sincerely trying to figure out in some detail how future AI developments might play out and impact society is a maddening, confusing activity that feels like trying to wring water from rocks.

1

2

46

Ajeya Cotra

@ajeya_cotra

1 year

AFAICT, there haven't been big wins historically from engaging in this kind of speculation about future technological developments.

3

1

45

Ajeya Cotra

@ajeya_cotra

1 year

At least, not something that would outperform some strategy like "try to generally learn about the technology and be close to the epicenter of its development, and be watchful and flexible and look for ways to help."

2

1

43

Ajeya Cotra

@ajeya_cotra

6 months

METR put out a standard way of defining tasks to evaluate agents that can interact with the world over a computer (e.g. AutoGPT etc): Currently being used by METR + UK AI safety institute + other orgs. Applicants to our RFP might find it useful!

Portable Evaluation Tasks via the METR Task Standard

metr.org

Ajeya Cotra

@ajeya_cotra

10 months

Experts disagree wildly on how powerful LLMs could get in the near future. Very excited to announce two new @open_phil RFPs for projects that could shed light on this. 1st, a call for realistic+difficult benchmarks for LLM agents (e.g. OpenAI's "GPTs"):

12

60

229

0

5

42

Ajeya Cotra

@ajeya_cotra

1 year

Strongly agreed on "To scientifically evaluate claims of humanlike and even superhuman machine intelligence, we need...better experimental methods and benchmarks." I really wish we were measuring suits of tasks more tightly correlated with real-world impacts.

Melanie Mitchell

@MelMitchell1

1 year

How do we know how smart AI systems are? | Science

37

210

678

1

4

40

Ajeya Cotra

@ajeya_cotra

1 year

2) I think we have reason to think (based on this kind of thoughtful speculation!) that *AI might go from pretty-powerful to super-powerful too quickly to react to in real time.*

2

42

Ajeya Cotra

@ajeya_cotra

1 year

@MikeWirth525 @jesswhittles @KatjaGrace by @DanHendrycks provides an overview of multiple risks, by @RichardMCNgo is a dive into the specific concern that powerful AI systems might be motivated to competently pursue goals their creators didn't intend.

1

40

Ajeya Cotra

@ajeya_cotra

7 months

If you have experience communicating thoughtfully, honestly, and compellingly about important topics, apply for the exec-level Comms Director role at @open_phil , even if you don't have a traditional PR background!

1

4

39

Ajeya Cotra

@ajeya_cotra

1 year

@jeremyphoward Agree, but historically we've been able to name specific massive capabilities gaps that could make us confident NNs were far from major real-world impacts (e.g. "they can't say coherent sentences," "they can't do math / logic"). It's a lot less clear now IMO.

4

0

39

Ajeya Cotra

@ajeya_cotra

11 months

Handy graphic from the blog post version of this paper on arguments for and against "explosive economic growth" from AI (aka at least tenfold increase in growth rates aka >= 30% YoY growth).

Tamay Besiroglu

@tamaybes

11 months

We assess if AI will accelerate economic growth by as much as growth accelerated during the industrial revolution, digging into growth theory, bottlenecks, feasibility of regulation, AI reliability/alignment, etc. Takeaway: acceleration looks plausible

22

108

475

0

4

39

Ajeya Cotra

@ajeya_cotra

1 year

Regulations on AI are currently very light compared to e.g. the aviation industry and I think the chance of catastrophic harm in the next 10y is >10x higher than an airplane crash (and our whole civilization would be on board).

1

37

Ajeya Cotra

@ajeya_cotra

1 year

If we make it through a transition to powerful AI, *most* of the value-added will probably come from people who are doing concrete, grounded things (ML research grounded in near-term problems, policy development grounded in near-term concerns, etc).

1

36

Ajeya Cotra

@ajeya_cotra

1 year

But I do it anyway because: 1) the stakes of potentially developing AI that can obsolete humans are high enough that I think *some* people in the world should be doing this activity even if it's in some sense low-ROI;

2

36

Ajeya Cotra

@ajeya_cotra

7 days

It's great to see this new cyber benchmark from @percyliang and co that we funded through our RFP! Funding ML is unfairly gratifying — we barely manage to get out the post announcing the grant by the time the grantee has a whole paper out 😂

1

4

36

Ajeya Cotra

@ajeya_cotra

5 months

The EU AI Office is looking for EU citizens with technical backgrounds to help implement the newly passed EU AI Act effectively. If you're interested, apply by Mar 27th!

Digital EU 🇪🇺

@DigitalEU

6 months

Interested in joining the 🇪🇺 AI Office as technical staff? Tune in to this online event to learn more about the work of the #AI Office, the latest job opportunities & get your questions answered. 🗓️ 20 March ⏰ 18:00 CET Register now 👇

1

39

83

0

12

33

Ajeya Cotra

@ajeya_cotra

1 year

@mattyglesias Hmm, I think there’s enormous upside from automating all the good things humans do to make everyone wealthier and healthier and more empowered; @KelseyTuoc wrote about that here:

The costs of caution

If you thought we might be able to cure cancer in 2200, then I think you ought to expect there’s a good chance we can do it within years of the advent of AI systems that can do the research work...

www.planned-obsolescence.org

2

3

36

Ajeya Cotra

@ajeya_cotra

1 year

I do think it's super unclear how far we have to go. Performance on a bunch of static benchmarks is high but it's really unclear to me how to translate that into real world impact; it seems *plausible* it's ~10y or so though, and the massive uncertainty scares me.

Arvind Narayanan

@random_walker

1 year

This is why I disagree with the "sparks of AGI" framing. There is no more reason to consider the current moment an inflection point than any previous moment in the history of ML (or technology). And we don't know how far we have left to go.

2

4

37

1

3

33

Ajeya Cotra

@ajeya_cotra

4 days

Loved this post () from @mattyglesias but IMO the nukes analogy misses how all-encompassing human-level AI would be: more like "industrial capacity" or "human capital" — important for war, but also everything else (and makes openness seem less weird).

2

1

38

Ajeya Cotra

@ajeya_cotra

2 months

We've rec'd $20M to agent benchmarks so far! 🎉 We're really excited to see what all our grantees build, and we're winding down the RFP to focus on new projects. New EOIs (300 words max) must be submitted *by Jul 26 at 11:59PM Eastern*:

[On hiatus] Request for proposals: benchmarking LLM agents on consequential real-world tasks | Open...

Update, 7/26/24: This RFP is now on hiatus; we are no longer accepting new expressions of interest. Update, 7/12/24: We will put this RFP on hiatus starting on 7/26/24 in order to focus on other...

www.openphilanthropy.org

Ajeya Cotra

@ajeya_cotra

5 months

Two updates on these funding opportunities: 1) We replaced our longer applications with a 300 word EOI 🎉 2) The broader RFP on studying+forecasting LLM impacts will be on hiatus after May 3 (the agent benchmarks RFP is still going strong!)

1

3

32

0

3

34

Ajeya Cotra

@ajeya_cotra

1 year

What are people's favorite practically-relevant ML benchmarks where SOTA LLMs do really poorly *even with chain-of-thought, tools, etc*?

5

0

31

Ajeya Cotra

@ajeya_cotra

1 year

New Planned Obsolescence post from my colleague Tom Davidson! In this post, Tom explains why most individual technologies don't accelerate economic growth, and why AI could be different.

Could AI accelerate economic growth?

Most new technologies don’t accelerate the pace of economic growth. But advanced AI might do this by massively increasing the research effort going into developing new technologies.

www.planned-obsolescence.org

1

3

31

Ajeya Cotra

@ajeya_cotra

5 months

Two updates on these funding opportunities: 1) We replaced our longer applications with a 300 word EOI 🎉 2) The broader RFP on studying+forecasting LLM impacts will be on hiatus after May 3 (the agent benchmarks RFP is still going strong!)

Ajeya Cotra

@ajeya_cotra

10 months

Experts disagree wildly on how powerful LLMs could get in the near future. Very excited to announce two new @open_phil RFPs for projects that could shed light on this. 1st, a call for realistic+difficult benchmarks for LLM agents (e.g. OpenAI's "GPTs"):

12

60

229

1

3

32

Ajeya Cotra

@ajeya_cotra

5 months

I think ultimately there will be a complicated and “boring” reconciliation here that involves being more open about architectures and training strategies, doing some stuff but not all stuff in secret, still trying to secure model weights, eating some costs on both sides, etc.

2

0

31

Ajeya Cotra

@ajeya_cotra

5 months

But IMO the prima facie case for AI risk makes reference to training choices. If we had a robust+mature science that showed that even OOD and tail behaviors are independent of training processes, we could ignore this. But we can't do much of that science outside companies.

3

0

28

Ajeya Cotra

@ajeya_cotra

5 months

I think a compelling safety case will often involve making statements about the training procedure and architecture. For example, it could matter whether models have persistent long-term memory, whether they’re trained with RL on metrics like medium-run profit, etc.

1

0

30

Ajeya Cotra

@ajeya_cotra

1 year

Really exciting development!

Ian Hogarth

@soundboy

1 year

I’m honoured to be appointed as the Chair of the UK's AI Foundation Model Taskforce. A thread on why I'm doing this and how you might be able to help us.

82

129

891

1

2

28

Ajeya Cotra

@ajeya_cotra

9 months

We've already gotten several exciting applications to these RFPs, and we want many more! We'll be hosting a webinar Wed at 10 am Pacific (in 2 days!) to help potential applicants better understand what we're looking for and how our process works:

Ajeya Cotra

@ajeya_cotra

10 months

Experts disagree wildly on how powerful LLMs could get in the near future. Very excited to announce two new @open_phil RFPs for projects that could shed light on this. 1st, a call for realistic+difficult benchmarks for LLM agents (e.g. OpenAI's "GPTs"):

12

60

229

0

4

29

Ajeya Cotra

@ajeya_cotra

1 year

Agree AI safety is highly interdsiciplinary. (I'd add infosecurity to this list.) A lot of work goes beyond academic disciplines too. E.g. for security it's not academic study of security that matters most; it's security practices in AI labs.

Jeremy Howard

@jeremyphoward

1 year

No one person has all the expertise needed to address AI safety.

24

74

370

2

4

29

Ajeya Cotra

@ajeya_cotra

1 year

Personally I have a >2.5% chance by 2029, something more like 15%? A lot of the disagreement here might be that I think it's unlikely we'll stay at "roughly human level" for very long, bc human-level AI can accelerate research into superhuman AI:

Continuous doesn’t mean slow

Once a lab trains AI that can fully replace its human employees, it will be able to multiply its workforce 100,000x. If these AIs do AI research, they could develop vastly superhuman systems in under...

www.planned-obsolescence.org

vitalik.eth

@VitalikButerin

1 year

@AISafetyMemes Very wide probability bars honestly. 95% confidence on timelines 2029-2200 for ASI, (I have significant probability mass on AGI staying roughly-human-level for a while) p(doom) around 0.1? Which of course makes it quite important to take AI risk seriously.

1

8

198

2

29

Ajeya Cotra

@ajeya_cotra

1 year

I would dearly love to not be in a world where I'm trying incredibly hard to finely interpret all this weak and ambiguous evidence we have about how AI might unfold.

2

3

29

Ajeya Cotra

@ajeya_cotra

1 year

Thanks so much for having me on the show, enjoyed this conversation!

Kevin Roose

@kevinroose

1 year

This week on Hard Fork: • Digging into the Surgeon General’s new warning about social media and adolescents • A conversation with the great @ajeya_cotra about AI’s existential risks • HatGPT returns!

8

2

30

3

0

29

Ajeya Cotra

@ajeya_cotra

5 months

But I don’t know exactly what the reconciliation would be on the substance, and I suspect a reconciliation that’s right on the substance might end up more pro openness than most companies would be willing to go for.

4

0

29

Ajeya Cotra

@ajeya_cotra

4 months

SOTA systems like ChatGPT are already more than just autoregressive LLMs. They're trained with RL on top of autoregressive (i.e. predict-the-next-token) loss, and the customer-facing product has all sorts of bells and whistles (often hand-engineered) on top of the trained model.

2

1

29

Ajeya Cotra

@ajeya_cotra

2 months

Great paper and post — focused on agent developers and application developers, but benchmarks that actually measure what they mean to and make it hard to cheat are just as indispensable for policymakers and safety researchers

Arvind Narayanan

@random_walker

2 months

Blog post with additional context about our new paper "AI Agents That Matter": - What does the term agent mean? Is it just a buzzword? - Do agents even work? - How does the poor reproducibility of agent evaluations compare with other areas of ML?

0

48

88

2

29

Ajeya Cotra

@ajeya_cotra

1 year

Staying close, paying attention, watching, and being flexible probably works better if we have the option. But crucial developments may happen so quickly that we don't have time to absorb them.

1

28

Ajeya Cotra

@ajeya_cotra

1 year

Curious for exceptions if people know about them! (Maybe bioweapons convention?)

4

1

28

Ajeya Cotra

@ajeya_cotra

5 months

I think externally-evaluated safety cases are super important, and to me it feels in tension with placing a lot of emphasis on avoiding sharing information about how SOTA systems are trained+built.

2

0

28

Ajeya Cotra

@ajeya_cotra

1 year

I don't think you need to be particularly confident about what concrete technologies become possible with far-superhuman AI in order to buy a story where humans lose control to AI.

2

1

27

Ajeya Cotra

@ajeya_cotra

1 year

I still think *most* people worried about AI risk shouldn't spend a lot of time engaging in detailed speculation about how AI developments might unfold.

1

26

Ajeya Cotra

@ajeya_cotra

1 year

In five years, I think a significant percentage of current MTurk tasks will probably be done by AI (would have to look at the tasks to get to a number). But I think that's short of automating all the most impactful things humans do (e.g. R&D, corporate decisionmaking, etc).

Shawn Presser

@theshawwn

1 year

This can be measured precisely: five years from now, what percentage of Mechanical Turk problems are being solved by AI? If your AI can do 100% of the solvable problems, it means you’ve achieved superintelligence, and you stand to earn all the money. My bet is, nowhere close.

11

3

33

3

1

26

Ajeya Cotra

@ajeya_cotra

20 days

Horizon is a great opportunity to get a foot in the door in AI policy. I especially hope ML PhD students and other technical folks consider it! Applications close Aug 30:

Horizon Fellowship ('25 Cohort)

THE HORIZON FELLOWSHIP HELPS EXPERTS IN EMERGING TECHNOLOGY LAUNCH THEIR CAREERS IN PUBLIC SERVICE. APPLICATIONS ARE NOW OPEN FOR THE 2025 COHORT. THE APPLICATION DEADLINE IS AUGUST 30TH, 2024....

jobs.ashbyhq.com

Horizon Institute for Public Service

@HorizonIPS

1 month

Apply to become a Horizon Fellow! Applications close Aug. 30. Curious what it is like to work on AI and biotech policy in DC? The Horizon Fellowship is a paid opportunity to work in a think tank, Congress or executive agency. Learn more here:

0

9

32

0

4

26

Ajeya Cotra

@ajeya_cotra

5 months

Notion 1: It's risky to widely disseminate information about architectures and training strategies for cutting-edge AI systems, because that advances AI R&D and means uncontrollably powerful AI is developed sooner with less time to prepare with safety+governance measures.

1

0

25

Ajeya Cotra

@ajeya_cotra

1 year

Really impressive! I know it's a temporary artifact that will be ironed out soon, but I actually like the wobbly trippy quality of current AI-generated videos. I'll be curious to see how this impacts the film industry.

Min Choi

@minchoi

1 year

AI just changed the game in filmmaking 📽️ Creators can now generate cinema quality short films, trailers, and teasers with 100% AI in just a few hours 🤯 10 favorite examples that will blow your mind:

855

896

5K

2

0

25

Ajeya Cotra

@ajeya_cotra

5 months

Notion 2: AI companies should have to make "safety cases" arguing that their SOTA systems are safe enough to train and deploy, and neutral third party experts should evaluate whether those safety cases pass muster.

1

0

25

Ajeya Cotra

@ajeya_cotra

1 year

Succinct blog post touching on this by @open_phil 's Tom Davidson here: Links to a dense but fascinating report on the topic.

Continuous doesn’t mean slow

Once a lab trains AI that can fully replace its human employees, it will be able to multiply its workforce 100,000x. If these AIs do AI research, they could develop vastly superhuman systems in under...

www.planned-obsolescence.org

1

3

25

Ajeya Cotra

@ajeya_cotra

1 year

I expect there to be lots of gains from synthetic data in domains where the model can come up with problems for itself where the solution is checkable but still difficult to find, e.g. stating theorems and attempting to prove them or making up programming challenges.

Arvind Narayanan

@random_walker

1 year

A popular view. I get it — learning w/o human knowledge seems implausible, even insulting. But let's wait and see. We already know this is false in some domains like game playing — synthetic data *alone* (self play) yields superhuman agents. In which other domains can it help?

30

27

233

0

24

Ajeya Cotra

@ajeya_cotra

1 year

This is a great opportunity, I especially encourage you to apply if you have a technical background and want to apply your skills to help shape AI policy!

Horizon Institute for Public Service

@HorizonIPS

1 year

Applications for the 2024 Horizon Fellowship are now open (deadline Sept 15th)! Do you have a background relevant to emerging tech policy? Interested in up to 2 years of funding to do impactful work on AI or biosecurity in DC? — Learn more here:

0

22

40

0

3

25

Ajeya Cotra

@ajeya_cotra

1 year

Informal adversarial collaboration testing GPT-4 with access to plug-ins (e.g. Wolfram Alpha) on math problems: Highlights that there's low-hanging fruit in training models to better use plug-ins

2

1

25