Eric Steinberger @EricSteinb Twitter profile | Pikagi

Pikagi

Eric Steinberger

@EricSteinb

7,425

Followers

503

Following

3

Media

168

Statuses

Writing code that writes code on a mission to build safe superintelligence | CEO/cofounder @magicailabs

Dug into my macbook

https://t.co/uo5bwpPijp

Joined September 2015

Don't wanna be here? Send us removal request.

Pinned Tweet

@EricSteinb

Eric Steinberger

5 months

I love my team a lot and sometimes it’s stressful but life has never been so fulfilling. If you want to build AGI on a small team of people who care a lot with thousands of GPUs, please apply :)

Tweet card media

Magic is an AI company that is working toward building safe AGI to accelerate humanity’s progress on the world’s most important problems.

@magicailabs

Magic.dev

5 months

We've raised $117M from @natfriedman and others to build an AI software engineer. Code generation is both a product and a path to AGI, requiring new algorithms, lots of CUDA, frontier-scale training, RL, and a new UI. We are hiring!

Tweet media one

35

94

684

19

11

190

Last Seen Profiles

@galery_basah10

@Masha0921Irin

@manie_nerp

@IsaNahilaS

@Jenaissasnte

@murrmason

@Susuoro

@lovestayjpg

@NYKagenda

@Oliver1163

@RiverWorksBFLO

@IStructE

@gm_stone

@IStructE

@Maria6589673846

@SoapLeopard

@SusannahBr87251

@dest_moon

@KeatonMcNutt

@eltiolab

@indowaria

@SMWebster

@Penningt1Denise

@RangersCare

@realdominochris

@LiviaBellona

@MEANYUNKl

@zptfay

@yukarinar

@LITCOACHBOOTHE

@r6a04kndwHxUGy0

@jay_harris101

@clownprincess52

@MattaHarbi

@kaede_akai00

@ianvanheusen

@EricSteinb

Eric Steinberger

4 months

Very excited to welcome @karpathy as Magic's latest investor!

40

48

1K

@EricSteinb

Eric Steinberger

1 year

When I was 15, I decided I'll dedicate my career to building superhuman AI. Today, some of the coolest people on Earth funded this endeavor with *$28 million*!

@magicailabs

Magic.dev

1 year

Announcing Magic's $23 million Series A to build an AI software engineer: @CapitalG (Alphabet), @natfriedman (fmr CEO GitHub), @eladgil , @Amplify , many others. We're hiring!

10

44

349

16

10

253

@EricSteinb

Eric Steinberger

5 months

Nat is a great sparring partner, coach and supporter. He has consistently pushed us to be even more ambitious while remaining practical. We are incredibly fortunate to have him as our major backer and now also as a board member at Magic.

@natfriedman

Nat Friedman

5 months

has trained a groundbreaking model with many millions of tokens of context that performed far better in our evals than anything we've tried before. They're using it to build an advanced AI programmer that can reason over your entire codebase and the

Tweet media one

79

193

2K

4

2

144

@EricSteinb

Eric Steinberger

1 year

AI with long-term memory! *A lot* of work left to do but happy to share a little more about what we've been up to. It's been incredibly fulfilling to work with a wonderful team and the trust of our backers towards this milestone. Thank you for the opportunity <3

@magicailabs

Magic.dev

1 year

Meet LTM-1: LLM with *5,000,000 prompt tokens* That's ~500k lines of code or ~5k files, enough to fully cover most repositories. LTM-1 is a prototype of a neural network architecture we designed for giant context windows.

48

185

1K

12

7

109

@EricSteinb

Eric Steinberger

5 months

Daniel is one of the most ambition-provoking people on Earth. Always pushes pushes pushes. That's how it should be.

@danielgross

Daniel Gross

5 months

Nat and I are delighted to be investing $100m in , which is building a superhuman software engineer. If a copilot generates $10b of revenue, how much is a colleague worth?

70

109

1K

4

3

85

@EricSteinb

Eric Steinberger

4 years

I believe in #opensource , and in making scientific results reproducable. Therefore, I am open-sourcing an implementation of DREAM, the state-of-the-art in Multi-agent model-free Deep RL. GitHub: #ai #deeplearning

This link will take you to a page that’s not on LinkedIn

@EricSteinb

Eric Steinberger

4 years

DREAM come true! :) Literally. @polynoamial , @adamlerer , and I developed an AI algorithm for multi-agent imperfect information games that's *100x more data-efficient* than the previous state-of-the-art. Check out the preprint (under NeurIPS review):

Tweet media one

1

10

49

2

11

59

@EricSteinb

Eric Steinberger

3 months

@altryne @magiclabs @karpathy @danielgross @natfriedman @polynoamial We don't have agents with 99.9% accuracy today -- that's just the goal!!

6

0

54

@EricSteinb

Eric Steinberger

5 months

fixed a bug I've been hunting for days! Yay

5

2

53

@EricSteinb

Eric Steinberger

4 years

DREAM come true! :) Literally. @polynoamial , @adamlerer , and I developed an AI algorithm for multi-agent imperfect information games that's *100x more data-efficient* than the previous state-of-the-art. Check out the preprint (under NeurIPS review):

Tweet media one

1

10

49

@EricSteinb

Eric Steinberger

3 months

@polynoamial @BRussellsimp Yup, email - in high school, I got a DeepMind research scientist as a mentor through an enormously long email (multiple A4 pages) of the shape "here's what I want to do to beat your algo, can you talk to me every 2 weeks to tell me when I'm stupid so I can get smart?"

2

2

40

@EricSteinb

Eric Steinberger

5 months

Thanks Noam! :)

@polynoamial

Noam Brown

5 months

2 years ago I invested in @magicailabs as my first-ever seed-round investment. I felt comfortable doing it because I'd worked with @EricSteinb for years and believed he'd succeed. I'm so impressed with the progress they've made and can't wait to see what they do next!

4

15

155

0

0

29

@EricSteinb

Eric Steinberger

1 year

@nextworddev @natfriedman So does the lecture

2

0

20

@EricSteinb

Eric Steinberger

5 months

Pretty damn cool that this worked out exactly as Nat hoped it would. Which other problems traditional approaches are stuck on could be solved with crowdsourcing young AI talent + startup thinking + prize pool?

@natfriedman

Nat Friedman

5 months

Ten months ago, we launched the Vesuvius Challenge to solve the ancient problem of the Herculaneum Papyri, a library of scrolls that were flash-fried by the eruption of Mount Vesuvius in 79 AD. Today we are overjoyed to announce that our crazy project has succeeded. After 2000

Tweet media one

2K

16K

70K

1

0

16

@EricSteinb

Eric Steinberger

2 years

Autism is a superpower, not a condition. Many of the most genuine, authentic, humble, and tenacious people I know are on the spectrum. Another one of my friends recently found out - loveliest dude ever, builds amazing things. Autism keeps you away from the prison of normalcy.

2

1

14

@EricSteinb

Eric Steinberger

1 year

Introducing @magicailabs - what if AI was a colleague, not a tool? We're starting with an AI software engineer, aiming to help developers build, ship, and review code faster and more effectively.

1

0

14

@EricSteinb

Eric Steinberger

5 months

@nealagarwal made a transformer

Tweet media one

2

0

13

@EricSteinb

Eric Steinberger

1 year

I can't thank our investors @natfriedman , @CapitalG , @ArthurB , @PrimordialAA , @eladgil , @Amplify , @polynoamial , et al. enough for trusting us with the resource we need to take a serious shot at building and productizing AGI, and for their helpful input.

2

0

13

@EricSteinb

Eric Steinberger

1 year

@john_lam It was just hard to make training stable and LTM-1 was the first time we succeeded at it. Had to make our own version of etc. Scaling should be less pain going forward now that we better understand the thing we made.

Tweet card media

Tensor Programs V: Tuning Large Neural Networks via Zero-Shot...

Hyperparameter (HP) tuning in deep learning is an expensive process, prohibitively so for neural networks (NNs) with billions of parameters. We show that, in the recently discovered Maximal Update...

1

0

13

@EricSteinb

Eric Steinberger

4 years

@jasonhickel And of good diets.

1

0

9

@EricSteinb

Eric Steinberger

4 years

Paul Erdős (1500+ papers published) on your left. Terence Tao (10 years old on this photo, now a Field's medallist) on your right. Pondering about a maths question together. This is adorable 😄

Tweet media one

0

0

10

@EricSteinb

Eric Steinberger

1 year

Mission: Build aligned and complete AI to accelerate humanity’s progress on the world’s most important problems.

2

0

9

@EricSteinb

Eric Steinberger

1 year

@Suhail Hi, I can give you 8 A100s on Lambda for a week if helpful

0

0

9

@EricSteinb

Eric Steinberger

1 year

We're hiring across ML Eng, GPU kernel Eng, product & design, full-stack, infra - you name it, we'd like to pay you for it. If you're hungry to build something truly awesome, please ping eric @magic .dev

0

0

8

@EricSteinb

Eric Steinberger

5 months

The coolest thing about it to me really isn't just that it happened (that's very very cool too!) but that it happened in exactly the way @natfriedman thought it would happen on a timeline that's roughly what I recall him saying it would be, like an item on a todo list

0

0

8

@EricSteinb

Eric Steinberger

1 year

@SebastianDeRo - thank you for leaving your previous CTO role to join us before there was any funding, and thank you for bringing your amazing team with you. The energy we have is uncomparable <3

2

0

8

@EricSteinb

Eric Steinberger

2 years

@ArthurB The race is on for another EA billionaire to emerge...

2

0

8

@EricSteinb

Eric Steinberger

1 year

@GaryMarcus @mustafasuleymn Hey @GaryMarcus ! I'd be happy to. - Random half of prompts open - other half secret & put out crypto hash - @goodside or some other credible judge Would need to define "LLM" as "any AI system" Btw - love your contribution to the regulation discussion, seriously thank you.

3

0

7

@EricSteinb

Eric Steinberger

1 year

@crazydonkey200 @arankomatsuzaki Why did you choose to compare with Adafactor for LMs? AdamW, as used in most parts of the paper, is a more competitive baseline. Afaik nobody is using factorization to train their models these days.

1

0

7

@EricSteinb

Eric Steinberger

3 years

@therealmjpoker @bp22 Who coded it? ;) Jokes aside, well done! Such incredible improvement in bp's game - you 1000% deserve this win and I hope you had a ton of side bets

1

0

7

@EricSteinb

Eric Steinberger

1 year

@magicailabs @CapitalG @natfriedman @eladgil @Amplify @PrimordialAA -- thank you for being the first one to say "yes" to funding us <3

0

0

6

@EricSteinb

Eric Steinberger

6 months

@jxmnop @NeelNanda5 Getting AI wrong has an existential (i.e. millions+ of years, all people) downside case. A brilliant person like @NeelNanda5 can singlehandedly move the needle on alignment, so this seems like a highly effective way for him to spend his time. Other issues matter too and most

0

0

2

@EricSteinb

Eric Steinberger

1 year

@natfriedman @CapitalG @ArthurB @PrimordialAA @eladgil @Amplify @polynoamial And thank you @bmoerzin , Johannes Heinrich, and @polynoamial for helping me become a good researcher in my school and college years! <3

0

0

5

@EricSteinb

Eric Steinberger

1 year

@samlakig @cis_female @aranku_ @mbpaulus we sponsor visas. I'd encourage anyone to apply!

2

0

5

@EricSteinb

Eric Steinberger

1 year

@jacobaustin132 @jeremyphoward @ESYudkowsky Hi! I should have been explicit, my bad. I don't have long tweets, so short here then long in next: GANs have no signal to get complexity beyond Discriminator capabilitiy into the Generator. LLMs do during pretraining, and RLHF doesn't try to remove it. More detail below

1

0

5

@EricSteinb

Eric Steinberger

11 months

@DynamicWebPaige Spirited Away is a beautiful movie but man was I scared shitless as a kid by... all of it

0

0

3

@EricSteinb

Eric Steinberger

11 months

@migtissera @magicailabs If you're having a bad day, I hope it gets better mate. Thanks for caring so much about our data quality! We're doing okay

1

0

4

@EricSteinb

Eric Steinberger

1 year

@kipperrii They love you back

0

0

4

@EricSteinb

Eric Steinberger

11 months

@migtissera @magicailabs We got more emails in the last 60 minutes than I would have thought we'd get in the first few days. We're targeting people who are creative about how to get tokens at low $ and effort. I think the $ is fair for many types of data sources.

1

0

4

@EricSteinb

Eric Steinberger

1 year

@ylecun To minimize perplexity on next-token prediction, it's helpful to predict further ahead, so it's likely that LLMs learn do this internally as an emergent ability. How else would they guess the next word ftera [Your paper] is [is], if not by thinking about what could come after?

2

1

4

@EricSteinb

Eric Steinberger

5 months

@Hersh_Desai ❤️❤️❤️❤️❤️❤️

0

0

4

@EricSteinb

Eric Steinberger

2 years

@karpathy Token hunter, I like it! Might use as job title

0

0

3

@EricSteinb

Eric Steinberger

8 months

@PrimordialAA @saranormous Yep, had 2 days worth of todos for the weekend and @aranku_ and I did ~80% of those 2 days in one day today. I did check the news every now and then though, I have to admit!

0

0

2

@EricSteinb

Eric Steinberger

1 year

@jacobaustin132 @jeremyphoward @ESYudkowsky - max learnable cognitive complexity for LLMs is a mix of the set of all forward generators and backward generators that generated the dataset (people, algorithms, randomness, ..), and the complexity of compressing them.

1

0

3

@EricSteinb

Eric Steinberger

2 years

@polynoamial Knowing how bad journalism gets in fields I know a lot about makes me wonder how bad it is everywhere else.

0

0

3

@EricSteinb

Eric Steinberger

11 months

@anammostarac @cis_female this is you

0

0

3

@EricSteinb

Eric Steinberger

6 months

@SarahChieng Small, focused team training frontier-scale foundation models to automate software engineering @magicailabs

1

0

3

@EricSteinb

Eric Steinberger

1 year

@SteverRobbins @magicailabs I believe automating work can shift societal incentives to focus on character over achievement. I know kind people with IQ 100 and a**holes with IQ 145. I'd like society to value kindness over achievement. Rough plan in next tweet.

1

0

3

@EricSteinb

Eric Steinberger

5 months

@typedfemale @_aidan_clark_ Writing them is fun. Debugging them is... yeah let's use CUDA

0

0

1

@EricSteinb

Eric Steinberger

7 months

@kipperrii Go Kip!:)

1

0

3

@EricSteinb

Eric Steinberger

1 year

@GergelyOrosz We are

0

0

3

@EricSteinb

Eric Steinberger

4 years

@mervyn_z @polynoamial It could, you are right! We just tested the pure form to see how well it learns from individual trajectories. Search is powerful and the direction Rebel takes is, in my opinion, likely closer to where the field is going to go than DREAM with no search.

0

0

3

@EricSteinb

Eric Steinberger

1 year

@Tim_Dettmers @VAribandi You say k/v norms - do you mean q/k? It makes more sense to me that k/v norms could struggle, but neither q nor k contribute to the residual state and you can get sharp attn with much lower logit values.

1

0

2

@EricSteinb

Eric Steinberger

1 year

@LeeLeepenkman @magicailabs LTM-1 is just a PoC model for giant ctx. We're now working on making it smart too (= model scale). First needed to iron out the architecture.

1

0

2

@EricSteinb

Eric Steinberger

1 year

@jacobaustin132 @jeremyphoward @ESYudkowsky - GANs don't ever learn beyond Discriminator's ability - RLHF RM provides no signal beyond RM's ability (like GAN's D), yep. HOWEVER: by same reasoning, RM signal can't encourage the LLM to unlearn skills obtained in pretraining. And in practise, people actively try to keep them

1

0

2

@EricSteinb

Eric Steinberger

1 year

@GaryMarcus But I trust Gary Marcus:) Keep up the great work

0

0

1

@EricSteinb

Eric Steinberger

1 year

@catherineols Ok:) Phew

0

0

2

@EricSteinb

Eric Steinberger

1 year

@alexgraveley "Bad people might use it" is the direction of the vector but the magnitude matters too. With encryption, risk seems predictably low. Would you say the same about nukes? There are many more examples like this - capped downside, large upside - but this doesn't represent AGI.

0

0

2

@EricSteinb

Eric Steinberger

10 months

@natfriedman @francoisfleuret @willdepue Way more, even if you only count post-GPT3

1

0

2

@EricSteinb

Eric Steinberger

5 months

@scholz_felix none tbh! Just started on the next thing

0

0

2

@EricSteinb

Eric Steinberger

6 months

@maxkriegers This is the first good-for-society social app I've seen in a long long time. Kudos @maxkriegers :)

0

0

2

@EricSteinb

Eric Steinberger

6 months

@miniapeur Noam Shazeer touched them

1

0

2

@EricSteinb

Eric Steinberger

1 year

@jacobaustin132 @jeremyphoward @ESYudkowsky Therefore, it's still in there and it's like RLHF never happened. Shoggoth and mask really does feel like the right analogy

1

0

2

@EricSteinb

Eric Steinberger

1 year

@McaleerStephen @cis_female 😶

0

0

2

@EricSteinb

Eric Steinberger

1 year

@RichardSocher @elonmusk I see you shitting on other (awesome) companies a lot lately. Why do you do this? Just build something great and make it big. If you're best you'll win. Don't try to make others small, especially not those who do good things well.

1

0

2

@EricSteinb

Eric Steinberger

4 years

@Govithinks I wish CVs would work. But yes, I also rejected people with CVs so beautiful that they could be art pieces and took in some people who didn't send a CV at all. Experience matters, but CVs don't reliably predict character and intelligence - the two things I care about most.

0

0

2

@EricSteinb

Eric Steinberger

3 months

@abiylfoyp good idea

0

0

2

@EricSteinb

Eric Steinberger

1 year

@SteverRobbins @magicailabs In this order 1. build & align ASI 2. protect against misuse thru work with gov et al 3. broad access @ everyone 4. pray it's a stable equilibrium Then: - let ASI do science in health, energy, .. - automate work. UBI; no poverty - Explore universe - Do whatever fulfills you

1

1

2

@EricSteinb

Eric Steinberger

2 years

@PrimordialAA Vienna is a gem too

1

0

2

@EricSteinb

Eric Steinberger

1 year

@kipperrii Solo-funding a *new* charity/project to get its thing off the ground. Early stage charity fundraising and grant writing sucks if you don't have the network.

0

0

2

@EricSteinb

Eric Steinberger

1 year

@Tim_Dettmers @VAribandi And you've seen those be problematic for outlier emergence?

1

0

1

@EricSteinb

Eric Steinberger

1 year

@stephenbalaban Congrats! Brrr**2

0

0

1

@EricSteinb

Eric Steinberger

5 months

@mschoening @mscccc @isamlambert oops

0

0

1

@EricSteinb

Eric Steinberger

2 years

@sama Not sure if that wouldn't actually slow them down. Being handed things for free does the opposite of preparing you to work hard to get them.

0

0

1

@EricSteinb

Eric Steinberger

1 year

@_builtin_katie @cis_female If you're super good at what u do we're flexible

0

0

1

@EricSteinb

Eric Steinberger

1 year

@SirMrMeowmeow @magicailabs 😶

0

0

1

@EricSteinb

Eric Steinberger

6 months

@kipperrii @eladgil Kip - is the intention of releasing this specific work to encourage the wider community to seek solutions?

0

0

0

@EricSteinb

Eric Steinberger

1 year

@DimitrisPapail @erijohnt @victorveitch @AnthropicAI It's interesting that you mind sharing it with Anthropic, but you don't mind sharing it with Slack itself. Why?

1

0

1

@EricSteinb

Eric Steinberger

11 months

@yimingdothan @magicailabs Interested too

0

0

1

@EricSteinb

Eric Steinberger

4 years

@pythiccoder @polynoamial @adamlerer Cheers! See you there

0

0

1

@EricSteinb

Eric Steinberger

4 months

@abiylfoyp Jim!!!!!

0

0

1

@EricSteinb

Eric Steinberger

5 months

@nearcyan @garrytan legend

0

0

1

@EricSteinb

Eric Steinberger

11 months

@sonikudzu @magicailabs GiB :)

1

0

1

@EricSteinb

Eric Steinberger

1 year

@senorgrubby @cis_female

Tweet card media

Magic is an AI company that is working toward building safe AGI to accelerate humanity’s progress on the world’s most important problems.

0

0

1

@EricSteinb

Eric Steinberger

1 year

@GuillaumeLample @GuillaumeLample I suspect the LR schedule plays into "even after 1T tokens the 7B model was still improving". A 7B run on 500B tokens could give clarity.

0

0

1

@EricSteinb

Eric Steinberger

1 year

@jacobaustin132 @jeremyphoward @ESYudkowsky Extremely good question. I currently think the answer is no for simple distillation and it would be capped at GPT4 smartness, but I'm not sure. This is not the case for humans

1

0

1

@EricSteinb

Eric Steinberger

2 years

@BusterFranken I think (almost) free is best. Maybe 5-10% of the average yearly income of a country and free for anyone who's in the bottom X% of earners. I observed that if it's completely free, people don't take it as seriously.

1

0

1

@EricSteinb

Eric Steinberger

7 months

@kipperriiii Did you scare them back?

1

0

1

@EricSteinb

Eric Steinberger

1 year

@DewanPandora @TokamakEnergy You're living the dream!!

0

0

1

@EricSteinb

Eric Steinberger

1 year

@yahorbarkouski @magicailabs Best IDE:)

0

0

1

@EricSteinb

Eric Steinberger

11 months

@MagnusHambleton startups do what startups do

1

0

1

@EricSteinb

Eric Steinberger

1 year

@0xDaksh @magicailabs WorkSans

0

0

1

@EricSteinb

Eric Steinberger

1 year

@mathemagic1an @AbrahamStarosta I knew you were going to hack on top of gpt4 today but this isn't the result I expected! :D

0

0

1

@EricSteinb

Eric Steinberger

2 years

@MaxThake Well, then someone's probably building web5 at the moment.

0

0

1

@EricSteinb

Eric Steinberger

2 years

@BusterFranken If data shows free is better, I'm for free! My reasoning was that people at free universities seem to take longer to complete their degrees (comparing Austria vs UK here)

1

0

1

@EricSteinb

Eric Steinberger

11 months

@cis_female Lol

1

0

1

@EricSteinb

Eric Steinberger

6 months

@eladgil Your music tweets are good - I remember you posted Ben Böhmer a while ago. Hadn't come across him before you posted and have been listening quite frequently since then

0

0

0

@EricSteinb

Eric Steinberger

1 year

@RichardSocher @elonmusk And for what it's worth, character has tons of users: Great team, product people use a ton, own the stack, moving fast, familiar interface for their audience.

Character.AI lets you create and talk to advanced AI - language tutors, text adventure games, life advice, brainstorming and much more.

1

0

1

@EricSteinb

Eric Steinberger

1 year

@abiylfoyp <3

0

0

1

@EricSteinb

Eric Steinberger

4 years

@Govithinks Alright, maybe that works. I haven't thought about this enough to know better but precisely because of that, I'll still remain humble in that I believe Warren Buffet and the slightly younger giants outsmart me in investing. Cool that you found a way to get a margin though!

1

0

1