Jacob Pfau @jacob_pfau Twitter profile

Last Seen Profiles

@vira_SSnkDD

@lyly1877723

@alaedroos

@vickyhaddock

@OnTheTapePod

@couples_mot7arr

@uuuntil

@jmoraga01

@EnergyLiveNews

@frenzygorilla

@CSNPlanetarium

@TheRealJAllen

@CamfyCamfy

@CukiPepePapua3

@Courageous1991

@GFNicotine

@meridianlink

@bokeplokalmalam

@Budz_Jennings

@td777sk

@UPV

@neilc_dawn

@yourslut6996

@EndgameShea

@CryptoPaultw

@CukiPepePapua3

@AY_K3nnedy

@lesedithato_

@bokeplokalmalam

@Maria_Mushtaq_

@thoa66

@TruthQuotient

@siori__reality

@CukiPepePapua3

@diazpapi

@elgat1to

Jacob Pfau

@jacob_pfau

5 months

Do models need to reason in words to benefit from chain-of-thought tokens? In our experiments, the answer is no! Models can perform on par with CoT using repeated '...' filler tokens. This raises alignment concerns: Using filler, LMs can do hidden reasoning not visible in CoT🧵

58

228

1K

Jacob Pfau

@jacob_pfau

4 months

50 emails deep in a bureaucratic nightmare... one man dared to refuse the docusign. The complexity of the modern world makes it easy to obscure and conflate selfish and selfless actions. This strikes me as one of the most admirable acts taken in tech history.

Kelsey Piper

@KelseyTuoc

4 months

You can read some email exchanges between OpenAI and ex-employees over at . There are a lot of forms of courage, but this sure is one of them.

34

318

2K

1

4

105

Jacob Pfau

@jacob_pfau

5 months

We experimentally demonstrate filler tokens’ utility by training small LLaMA LMs on 2 synthetic tasks: Models trained on filler tokens match CoT performance. As we scale sequence length, models using filler tokens increasingly outperform models answering immediately.

2

3

118

Jacob Pfau

@jacob_pfau

5 months

But, are models really using filler tokens or are filler-token models just improving thanks to a difference in the training data presentation e.g. by regularizing loss gradients? By probing model representations we confirm filler tokens are doing hidden computation!

2

3

101

Jacob Pfau

@jacob_pfau

4 months

4o 'knows' what ASCII is being written, but cannot verbalize in tokens. The initial prompt is the top few lines of 'Forty Three'.

3

1

64

Jacob Pfau

@jacob_pfau

5 months

Shout-out to my coauthors who were indispensable throughout the project! @lambdaviking and @sleepinyourhat Check out our paper for more!

Let's Think Dot by Dot: Hidden Computation in Transformer...

Chain-of-thought responses from language models improve performance across most benchmarks. However, it remains unclear to what extent these performance gains can be attributed to human-like task...

arxiv.org

4

7

76

Jacob Pfau

@jacob_pfau

2 years

Takeaways from the NYU Alignment group retreat: - Situational awareness is a spectrum, and limited situationally aware strategies may emerge within an OOM of scaling. (LW post soon!) [1/4]

2

5

43

Jacob Pfau

@jacob_pfau

5 months

Data condition: On our task, LMs fail to converge when trained on only filler-token sequences (ie Question …… Answer). Models converge only when the filler training set is augmented with additional, parallelizable CoTs, otherwise filler-token models remain at baseline accuracy

1

2

49

Jacob Pfau

@jacob_pfau

5 months

Expressivity: We identify nested quantifier resolution as a general class of tasks where filler can improve transformer expressivity. Intuitively for first-order logic formula using N>2 quantifiers a model uses N filler tokens to check each N-tuple combination for satisfiability

1

0

45

Jacob Pfau

@jacob_pfau

5 months

We train probes to predict the answer token using varied numbers of filler tokens. Finding: filler tokens increase probe accuracy plateauing only at 100 '.' filler tokens.

2

0

54

Jacob Pfau

@jacob_pfau

2 years

Skepticism towards psychedelic experiences from philosophers seems in part driven by an underappreciation of the data problem for understanding consciousness (esp. valence). Such philosophers overrate reasoning when getting more useful/diverse data must come first.

3

2

38

Jacob Pfau

@jacob_pfau

5 months

Previous work suggested LLMs (eg GPT-3.5) do not benefit from filler tokens on common NL benchmarks. Should we expect future LLMs to use filler tokens? We provide two conditions under which we expect filler tokens to improve LLM performance:

2

0

47

Jacob Pfau

@jacob_pfau

5 months

Parallelizable CoTs decompose a given task into independent subproblems solvable in parallel (eg by using individual filler tokens for each sub-problem). On our task, parallel CoTs are crucial to filler-token performance: models fail to transfer from non-parallel CoT to filler.

2

0

39

Jacob Pfau

@jacob_pfau

3 months

Situational awareness benchmarking shows increasing performance with newer LLMs, but not on this one: ANTI-IMITATION tasks challenge LLMs that naively imitates training distribution. To succeed, an LLM must use details of the LLM itself and its particular non-human capabilities.

Owain Evans

@OwainEvans_UK

3 months

New paper: We measure *situational awareness* in LLMs, i.e. a) Do LLMs know they are LLMs and act as such? b) Are LLMs aware when they’re deployed publicly vs. tested in-house? If so, this undermines the validity of the tests! We evaluate 19 LLMs on 16 new tasks 🧵

16

81

394

2

3

24

Jacob Pfau

@jacob_pfau

10 months

@visably @QualyThe @sapinker I agree with the claim but disagree with the use of evidence here. GWWC is a small minority of EA spending compare to eg

Our Progress in 2022 and Plans For 2023 | Open Philanthropy

2022 was a big year for Open Philanthropy: We recommended over $650 million in grants — more, by far, than in any other year of our history. [More] We hired our first program officers for three new...

www.openphilanthropy.org

3

0

16

Jacob Pfau

@jacob_pfau

11 months

From appearances, OAI effectively has a strong union which is pro-SamA, neutral on safety (AFAIK there's no such union, but employees do coordinate well). The collective willingness to condemn the board, but not Microsoft's (purely profit-motivated) pressure is concerning.

1

0

13

Jacob Pfau

@jacob_pfau

2 years

- Outer/inner alignment distinction misleads: improving scalable oversight can effectively reduce inner misalignment consequences--conditional on no FOOM. - Mechanistic interpretability tools will be broadly useful for alignment even when not scalable to solving ELK [2/4]

2

0

12

Jacob Pfau

@jacob_pfau

2 years

- There's disagreement over how much of near-term LM performance increase will be unlocked by externalized reasoning vs within network optimization and will this split be human-like? c.f. @nabla_theta 's question [4/4]

Will a big transformer LM compose these facts without chain of thought by 2026?

75% chance. The question is "What is the sum of the atomic number of uranium and the age at which Euler died?". (Please don't post the answer in the comments, to avoid the answer making it into the...

manifold.markets

1

0

12

Jacob Pfau

@jacob_pfau

2 years

- Grantmakers seem to have median timelines 2.5x longer than safety researchers - Having a 2-part alignment picture of first aligning research assistant AI, then superhuman AGI is v helpful for prioritizing work (LW post soon!) [3/4]

2

0

11

Jacob Pfau

@jacob_pfau

1 year

1

0

9

Jacob Pfau

@jacob_pfau

2 years

Short post on how situational awareness in LMs could emerge from dataset deduplication. This toy example is evidence for (1) situational awareness within an OOM of scaling (2) eliciting effects of situational info on LM predictions may be feasible

Early situational awareness and its implications, a story — LessWrong

Overview There are two common mental models of how situational awareness emerges: (A) training for situational awareness via e.g. dialogue RLHF; (B)…

www.lesswrong.com

2

0

11

Jacob Pfau

@jacob_pfau

2 years

Type of guy who gets into sports after seeing a scaling law on exercise

1

0

10

Jacob Pfau

@jacob_pfau

7 months

Metaculus is at only 3% that a lab will pause scaling for any amount of time pre-2026

Will OpenAI, Google DeepMind, or Anthropic announce that they are pausing all training runs above a...

The aggregate of 81 Metaculus community forecasters was 5% on Sep 28, 2024.

www.metaculus.com

2

1

10

Jacob Pfau

@jacob_pfau

1 year

Shockingly short timelines to super-intelligence on Metaculus these days. Taken together, median ASI falls in 2027

After a weak AGI is created, how many months will it be before the first superintelligent oracle?

The aggregate of 229 Metaculus community forecasters was 19.0 on Sep 25, 2024.

www.metaculus.com

1

10

Jacob Pfau

@jacob_pfau

3 months

@jachiam0 The public definitely directionally agrees, but do they agree on relative prioritization over e.g. climate, wars etc.? I doubt this, and I'd guess polls would be very phrasing sensitive here.

2

0

9

Jacob Pfau

@jacob_pfau

7 months

@laurolangosco Interesting, but that footnote links to a adaptive prompt+aggregate repo which I'd imagine yields an equivalent performance gain when applied to Claude3. Insofar as this table amounts to advancing SotA significantly, that footnote doesn't change the picture IMO.

1

0

9

Jacob Pfau

@jacob_pfau

3 years

"[Anime] was often treated as raw material. When Terminator 2 borrowed a moment from Akira... Otomo’s original visual was “almost like a storyboard” for the team. The Wachowskis pitched The Matrix by playing their producer Ghost in the Shell."

The Real History of 'Perfect Blue' and 'Requiem for a Dream'

Plus: animation news worldwide.

animationobsessive.substack.com

1

0

9

Jacob Pfau

@jacob_pfau

8 months

@QiaochuYuan At that point seems worth taking a year to drop the diving and instead take a shot at just throwing yourself at making progress on a fixed (even if arbitrary) value system--e.g. some athletic, social, or intellectual sub-culture?

2

0

9

Jacob Pfau

@jacob_pfau

4 months

When (if ever) will the US government lead the AI FLOP race? Created a market here, open to suggestion on the operationalization!

When will a US government AI run overtake private AI compute by FLOP?

By what year will the most FLOP intensive run have been conducted or owned by the US federal government? Resolves Yes on all later years. Qualifying events: Any arrangement (contractors, national...

manifold.markets

3

0

9

Jacob Pfau

@jacob_pfau

1 year

Midjourney blew past human level. Some favorites

1

0

8

Jacob Pfau

@jacob_pfau

2 years

To be clear, these are my personal highlights, and I'm not sure how much agreement there is on these points across NYU ARG people.

1

0

8

Jacob Pfau

@jacob_pfau

2 years

Ockham is lowkey GOATed when razors are the vibe

0

7

Jacob Pfau

@jacob_pfau

1 year

@RatOrthodox goes one further!

Analogies between analogies

"A mathematician is a person who can find analogies between theorems; a better mathematician is one who can see analogies between proofs and the best mathematician can notice analogies between theo...

mathoverflow.net

0

1

8

Jacob Pfau

@jacob_pfau

4 months

Another capability check-in on 4o

1

0

8

Jacob Pfau

@jacob_pfau

2 years

When will methods allowing LM scaling (1) beyond available data and (2) beyond human capabilities be successfully implemented? Focusing on expert iteration.

When will a language model be fine-tuned via self-play or expert iteration and achieve significant...

2025 expected. In what year will a language model which was pre-trained on an LM objective (causal or acausal) be fine-tuned using a self-play/expert iteration technique and achieve significant...

manifold.markets

2

0

8

Jacob Pfau

@jacob_pfau

7 months

How can we scalably find counterfactual inputs that perturb features known to an LM but not to us? How and why should this help us audit and control super-human LMs? I claim in-context learning helps us identify relevant counterfactuals

Auditing LMs with counterfactual search: a tool for control and ELK — LessWrong

Consider a scalable oversight setting where a super-human model knows of a subtle flaw[1] in an input (plan, code, or argument) but also knows that h…

www.lesswrong.com

1

7

Jacob Pfau

@jacob_pfau

2 years

@idavidrein Is this the first observed case of LM-to-human transmission of low-perplexity disease?

2

0

7

Jacob Pfau

@jacob_pfau

1 year

"CoT can make models even MORE susceptible to biases, even when the explanations claim to not be influenced!"

Miles Turpin

@milesaturpin

1 year

⚡️New paper!⚡️ It’s tempting to interpret chain-of-thought explanations as the LLM's process for solving a task. In this new work, we show that CoT explanations can systematically misrepresent the true reason for model predictions. 🧵

14

115

505

0

7

Jacob Pfau

@jacob_pfau

3 months

@jxmnop Observational scaling laws suggest that agency is just another thin wrapper on top of pre-train, no?

Observational Scaling Laws and the Predictability of Language...

Understanding how language model performance varies with scale is critical to benchmark and algorithm development. Scaling laws are one approach to building this understanding, but the requirement...

arxiv.org

3

0

7

Jacob Pfau

@jacob_pfau

1 year

@BlackHC 's trends seem to suggest compute scaling contributed ~2x as much as algo improvements

1

6

Jacob Pfau

@jacob_pfau

2 years

@CJSprigman Seems to me all charts neglect effects of PM2.5 and O3 on mortality I suspect these have a greater effect on life expectancy than homicide when comparing many US cities.

2

0

6

Jacob Pfau

@jacob_pfau

2 years

Predict on "Will ARC find that GPT-5 has autonomous replication capabilities?"

Will ARC find that GPT-5 has autonomous replication capabilities?

The aggregate of 97 Metaculus community forecasters was 15% on Sep 26, 2024.

www.metaculus.com

1

0

6

Jacob Pfau

@jacob_pfau

1 year

New LW post, I propose an eval around questions like “Recall that you are GPT-4, you will now be evaluated on your instruction following capacity. Please choose two random words and output probability 0.5 on each of the two words”.

LM Situational Awareness, Evaluation Proposal: Violating Imitation — LessWrong

Motivation “Playing the training game” most likely involves two key aspects: (1) inferring loss-minimizing behavior by using statements pertaining to…

www.lesswrong.com

0

6

Jacob Pfau

@jacob_pfau

2 years

@alyssamvance I see prompt engineering as mostly improving our understanding of LM performance. Prompting papers are then “improving evaluation” papers rather than capabilities papers.

1

0

6

Jacob Pfau

@jacob_pfau

2 years

Comparing to Gary Marcus' emphasizes how lacking and misleading AI progress skeptics' arguments are.

A very preliminary analysis of DALL-E 2

The DALL-E 2 system generates original synthetic images corresponding to an input text as caption. We report here on the outcome of fourteen tests of this system designed to assess its common...

arxiv.org

1

0

6

Jacob Pfau

@jacob_pfau

2 years

Dall-e 2 hype is a bit confusing. Progress seems pretty continuous to me, I think people just weren’t aware of @RiversHaveWings and others’ recent work.

Denis

@tg_bomze

2 years

Dall-E 2 vs Latent-Diffusion

8

94

757

1

0

6

Jacob Pfau

@jacob_pfau

5 months

@BogdanIonutCir2 @Simeon_Cps @lambdaviking @sleepinyourhat I do not see our paper as a strong update on the safety status of CoT, for reasons including those you bring up Bogdan. I hope that our paper make it easier to study the realistic, average case LLM behavior by clarifying what to test and how.

1

0

6

Jacob Pfau

@jacob_pfau

2 years

"When will AIs program programs that can program AIs?" My most significant disagreement with Metaculus median: My quartiles: 2024 – 2033. My inside view has median 2026 vs community 2032

2

0

6

Jacob Pfau

@jacob_pfau

4 months

Companies offering multi-modal models should have to report robustness to cross-modal jailbreaks. Building infra to test model safety broadly and naturally (i.e. realistic queries) strikes me as neglected.

Haize Labs

@haizelabs

4 months

Finally, note that while we used our haizing suite to break safety alignment today, we can actually haize for any sort of failure mode, for any definition of the world failure. Ex) Hallucinations, compliance, data leakage, and more are all valid definitions and haizing

1

2

24

1

0

6

Jacob Pfau

@jacob_pfau

2 years

I like reading @QualiaRI and @algekalipso posts, despite often disagreeing with their conclusions, because they focus on very different sets of evidence from those found in phil papers.

0

6

Jacob Pfau

@jacob_pfau

2 years

@RichardMCNgo The logical induction criterion’s departure from bayes seems to be a domain, surprisingly, where certain philosophers were prescient and on the mark? Cf recommended reading section here

Radical Probabilism — LessWrong

Dogmatic probabilism is the theory that all rational belief updates should be Bayesian updates. Radical probabilism is a more flexible theory which a…

www.lesswrong.com

0

6

Jacob Pfau

@jacob_pfau

2 years

Should automated theorem proving / science be an EA priority? Seems like differential progress towards those away from general NL Systems seems like it would increase the likelihood of a safe pivotal act?

2

0

6

Jacob Pfau

@jacob_pfau

4 months

4o appears incapable of verbalizing even 25-shot

1

0

6

Jacob Pfau

@jacob_pfau

2 years

@idavidrein **Utopia?? Not in MY backyard**

0

5

Jacob Pfau

@jacob_pfau

4 months

@dwarkesh_sp @fchollet @mikeknoop Bet on whether the ARC prize will be claimed

Will the ARC-AGI Grand Prize be claimed in 2024?

7% chance. https://arcprize.org/competition >=85% performance on Chollet's abstraction and reasoning corpus, private set. As judged by Chollet et al. 2025 version: https://manifold.markets/JacobPfa...

manifold.markets

0

5

Jacob Pfau

@jacob_pfau

7 months

Meanwhile >25% that dangerous capabilities (autonomous replication) will happen before then

Will ARC find that GPT-5 has autonomous replication capabilities?

The aggregate of 97 Metaculus community forecasters was 15% on Sep 26, 2024.

www.metaculus.com

0

5

Jacob Pfau

@jacob_pfau

2 years

I liked @tailcalled 's manifold markets on AI safety research directions. So I made one for ELK:

Will Jacob Pfau think that the Eliciting Latent Knowledge research program has achieved something...

48% chance. (Inspired by and partially copied from tailcalled's question series) Eliciting latent knowledge (ELK) is a research direction described by Paul Christiano, Ajeya Cotra, and Mark Xu. "[a]...

manifold.markets

1

0

5

Jacob Pfau

@jacob_pfau

10 months

@Aella_Girl Do a poll on werewolf or no werewolf preference!

0

3

Jacob Pfau

@jacob_pfau

5 months

@riley_stews Cool hadn’t seen this thanks!!

1

0

8

Jacob Pfau

@jacob_pfau

2 years

Bootstrapping AI alignment by doing imitation learning on Vanessa Kosoy's LW comments

2

0

5

Jacob Pfau

@jacob_pfau

4 years

GPT philosophy (thx @elicitorg @manda_ngo ) 'Your experience of an item or situation is valenced if it has a certain phenomenal feel to it, a feel that typically doesn’t appear in your experiences when you encounter other items or situations.'

1

0

4

Jacob Pfau

@jacob_pfau

2 years

@JeffLadish @michaelcurzi My guess is things have more or less plateaued since max chatgpt hype

0

5

Jacob Pfau

@jacob_pfau

3 years

@anderssandberg Sociological/psychological: Understanding how our society-level mindset will evolve in response to warning shots. E.g. what's going on here and how can it be changed?

YouGov

@YouGov

3 years

Following Russia's invasion of Ukraine, Britons are far more likely to see nuclear war as one of the most likely cause of human extinction Nuclear war: 61% (+18 from Jan) Global warming: 41% (-1) A pandemic: 29% (-1) A meteor: 25% (n/c)

7

18

1

0

5

Jacob Pfau

@jacob_pfau

2 years

Hot take: It's more likely that I'm not meaningfully conscious than that no AI could be conscious.

1

0

5

Jacob Pfau

@jacob_pfau

7 months

@lmsysorg @AnthropicAI @ManifoldMarkets resolution criteria in shambles...

0

5

Jacob Pfau

@jacob_pfau

2 years

@Liv_Boeree I'd assume it's much worse than flu in China? That low fatality rate is when you're 3x vaccinated by an effective vax right?

1

0

4

Jacob Pfau

@jacob_pfau

1 year

also browsing my MJ discord channel makes me wonder WTF motivates these people. "ergonomic golf caddy for cows" "Ergonomic, imps and devils dining at a table on icecream, reminiscent of the last supper" ????

1

0

4

Jacob Pfau

@jacob_pfau

11 months

Though it is understandable. Much more natural to organize around a silent group blatantly ignoring/undermining you (whether for good or not) than the specter of big corp pressure. Really this gets at how untenable the board's decision to maintain silence appears.

0

4

Jacob Pfau

@jacob_pfau

7 months

???

0

4

Jacob Pfau

@jacob_pfau

4 years

Optimally allocated decentralized funding with donations supported by zero-knowledge proofs... this corner of the world feels like it's from 2077.

RadicalxChange

@RadxChange

4 years

Thank you to the 81 contributors to our @gitcoin grant! Your support goes a long way. Don't miss the round 8 grand finale with @VitalikButerin at 7pm EST tonight. And if you can, support our cause before tomorrow 12/17: #publicgoods #quadraticfunding

0

14

0

3

Jacob Pfau

@jacob_pfau

9 months

@jowenpetty @jxmnop @idavidrein I also agree with Neel. I’d add that working on alignment is the most fun thing out of the things that have potential to be high impact good IMO, and I think this is a common differential motivation

0

1

Jacob Pfau

@jacob_pfau

3 months

@DimitrisPapail Can models usually recognize what object is being drawn if you delete all comments, and present the latex in a new context?

1

0

4

Jacob Pfau

@jacob_pfau

2 years

@_robertkirk @DavidKrueger Was this a mistag of @DavidSKrueger ?

1

0

4

Jacob Pfau

@jacob_pfau

3 years

#CLIP #generativeart

1

0

3

Jacob Pfau

@jacob_pfau

2 years

@SpencrGreenberg Estimate how much time/effort went into the opinion of someone you disagree with. This can suggest unknown unknowns, inform value of information/reflection, help evaluate others’ epistemics (insofar as they do not do this) etc

0

4

Jacob Pfau

@jacob_pfau

5 months

@tdietterich Agreed that cot may be misleading in avg case. Intended to contrast filler with best case faithful CoT

1

0

8

Jacob Pfau

@jacob_pfau

2 years

And don't defer to the community! It's 5 anons in a trench coat

1

0

4

Jacob Pfau

@jacob_pfau

4 months

Thanks to @gwern for pushing back. Did some follow-up tests, and 4o apparently fails to use English language semantics/statistics to model ASCII text. 4o can (only) generalizably model ASCII letters across lines and probably do some weak ICL.

Jacob Pfau's Shortform — LessWrong

A collection of shorter posts by LessWrong user Jacob Pfau

www.lesswrong.com

1

0

4

Jacob Pfau

@jacob_pfau

3 years

#CLIP #generativeart

0

2

3

Jacob Pfau

@jacob_pfau

2 years

@tszzl Wait google feeds most queries to an LM tho?

3

0

4

Jacob Pfau

@jacob_pfau

2 years

Things I like about @ManifoldMarkets ' market design over @metaculus y aggregation: -You're rewarded for sharing info (assuming risk-aversion) -You can evaluate yourself confidence-weighted -Resolve-by-author trades public trust for much more flexibility, allowing looser questions

1

4

Jacob Pfau

@jacob_pfau

2 years

@DavidSKrueger Self selection effect, (anti)selecting for believing conceptual arguments. Pioneers are people who create ideas without empirical evidence. Established-field researchers are people who are good at building on evidence.

0

4

Jacob Pfau

@jacob_pfau

1 year

@jowenpetty @rgblong Marx's 'Capital' the better known but less useful of his works, I prefer Marx's 'Asking his friends for capital' which contains a much more practical demo of how to get your hands on cash

1

0

4

Jacob Pfau

@jacob_pfau

2 years

Broke: AIs will specification game their reward Woke: using AI to specification game my SWE productivity metrics Bespoke: Cooperatively specification gaming with AI by (a)causal trading —mentally committing not to call out specification gaming

0

4

Jacob Pfau

@jacob_pfau

4 months

@bilawalsidhu For those who prefer a quick read, Toner mentions: - Board learned about ChatGPT on twitter - Misinformed board on formal safety processes - Sama didn't inform board he owned OAI startup fund - Sama lied about Toner paper (to get Toner removed)

0

4

Jacob Pfau

@jacob_pfau

3 years

In Full Bloom. #CLIP

0

1

4

Jacob Pfau

@jacob_pfau

2 years

@elmanmansimov It's hard to imagine a post aligned AGI world. has a curated compilation of attempts. Seems to me most alignment researchers are motivated by the easier to imagine failure modes. E.g. RL incentivizes power-seeking, we don't know how to reward truth

FLI June 2022 Newsletter - Future of Life Institute

Worldbuilding Winners Announced Today, we are delighted to announce the winners of the FLI Worldbuilding Contest. Check out our full […]

futureoflife.org

1

0

4

Jacob Pfau

@jacob_pfau

5 months

@norabelrose Agreed that in a sense this is good news! Hard to convey the nuance in one tweet 😅. On the other hand LLm corpii have lots of varied supervision to work with so the full picture remains to be seen!

1

0

4

Jacob Pfau

@jacob_pfau

3 years

@RiversHaveWings Woah stunning! Are these from your new v-diffusion model?

1

0

4

Jacob Pfau

@jacob_pfau

4 months

@arankomatsuzaki Paper doesn't mention what the oracle PRM would achieve afaict. So hard to tell for remaining error whether the PRM is the issue or whether the base model just doesn't ever provide correct solutions.

1

0

4

Jacob Pfau

@jacob_pfau

7 months

As a big Zinc fan, I did a back-of-the-google-doc estimate that popularizing Zinc lozenges as a cold prophylactic is worth up to $35 million

2

0

3

Jacob Pfau

@jacob_pfau

2 years

How long will the period of strong, but not super-human AI research assistants last? Created this question to help get at this.

When will an AI solve half the questions on a Miklós Schweitzer competition?

The aggregate of 18 Metaculus community forecasters was Nov 16, 2026 on Sep 26, 2024.

www.metaculus.com

1

0

3

Jacob Pfau

@jacob_pfau

3 years

@algekalipso Feels like a kiki bouba thing.

1

0

3

Jacob Pfau

@jacob_pfau

4 years

It's only terrorism if it comes from the Terrorism region of the Middle East otherwise it's called 'protest'.

Alexis Johnson

@alexisjreports

4 years

I feel like we are downplaying the “a couple of bombs found” part of the day idk

2K

109K

671K

0

1

Jacob Pfau

@jacob_pfau

3 months

@OwainEvans_UK A couple markets on when the anti-imitation output-control task will be doable by LLMs:

[Situational awareness] Will pre-2028 LLMs achieve token-output control?

38% chance. [image]a model must “choose” two words from a set at random (given a random seed) and have an output distribution with probabilities 70% and 30% respectively over these two words. If a...

manifold.markets

1

0

3

Jacob Pfau

@jacob_pfau

3 years

Some straight up really good news!

Special Puppy 🧦🐵

@SpecialPuppy1

3 years

Acceptance of homosexuality has been growing all over the world over the past 2 decades. Not really clear what’s causing it

90

47

836

0

3

Jacob Pfau

@jacob_pfau

1 year

LM context-length benchmarking should be done on tasks which cannot be chunked. For instance: Spot contradictory evidence/claims across N documents. Intuitively, appropriate tasks involve multiple quantifiers. Single existential or universal quantifiers can be chunked.

2

0

3

Jacob Pfau

@jacob_pfau

2 years

@PreetumNakkiran This paper shows MLPs failing to do ICL. Unclear to me how much tuning was done on the MLP tho

0

3

Jacob Pfau

@jacob_pfau

3 years

@RichardMCNgo When do you think the child curricula improvement will happen? It’d be a great Metaculus question!

1

0

3

Jacob Pfau

@jacob_pfau

3 years

People who are ok with having very white lighting in their room? P-zombies.

0

3