Karol Hausman @hausman_k Twitter profile

Pinned Tweet

Karol Hausman

4 months

🚨 Big news 🚨 Together with a set of amazing folks we decided to start a company that tackles one of the hardest and most impactful problems - Physical Intelligence In fact, we even named our company after that: or Pi (π) for short 🧵

45

46

549

Last Seen Profiles

@ansari2487

@FiturZindgi

@hannah_teufel

@w21027

@JBVample

@pengen_stw

@mosu_oyuba

@BandarStw

@pengen_stw

@glang1

@jaythornton000

@AbbaA3872

@CadenDespair

@as5Txbk83fsOSa1

@SeneddCymru

@ATNfestival

@Lord_Snerts

@LittleShell03

@DanielHeithorn

@JaviNieves

@PatrikNacher

@robinavni

@SandyZarat82044

@bijans

@YVM1946

@007_feroz

@HazzaniArt

@hijabjilbab1

@algaecorner

@Gumtree

@aquarELLAz

@MasMallowZzz

@us4gi0

@thevoxval

@osiris_trinity

@Shaa__ee

Karol Hausman

@hausman_k

2 years

Introducing RT-1, a robotic model that can execute over 700 instructions in the real world at 97% success rate! Generalizes to new tasks✅ Robust to new environments and objects✅ Fast inference for real time control✅ Can absorb multi-robot data✅ Powers SayCan✅ 🧵👇

62

553

2K

Karol Hausman

@hausman_k

2 years

New large language model released by @GoogleAI . Take a look, it's pretty hard to believe. It can explain jokes:

Google AI

@GoogleAI

2 years

Introducing the 540 billion parameter Pathways Language Model. Trained on two Cloud #TPU v4 pods, it achieves state-of-the-art performance on benchmarks and shows exciting capabilities like mathematical reasoning, code writing, and even explaining jokes.

76

1K

4K

55

301

2K

Karol Hausman

@hausman_k

4 years

It's just overfitting, they said. It just memorizes the data, they said.

23

184

2K

Karol Hausman

@hausman_k

2 years

Super excited to introduce SayCan (): 1st publication of a large effort we've been working on for 1+ years Robots ground large language models in reality by acting as their eyes and hands while LLMs help robots execute long, abstract language instructions

19

287

1K

Karol Hausman

@hausman_k

2 years

Have you ever “heard” yourself talk in your head? Turns out it's a useful tool for robots too! Introducing Inner Monologue: feeding continual textual feedback into LLMs allows robots to articulate a grounded “thought process” to execute long, abstract instructions 🧵👇

24

169

899

Karol Hausman

@hausman_k

1 year

Very exited to announce our largest deep RL deployment to date: robots sorting trash end-to-end in real offices! (aka RLS) This project took a long time (started before SayCan/RT-1/other newer works) but the learnings from it have been really valuable.🧵

18

170

855

Karol Hausman

@hausman_k

2 years

Me: Nobody understands my sense of humor AI: I got you.

Karol Hausman

@hausman_k

2 years

New large language model released by @GoogleAI . Take a look, it's pretty hard to believe. It can explain jokes:

55

301

2K

17

132

784

Karol Hausman

@hausman_k

6 months

AGI has been achieved internally

47

12

705

Karol Hausman

@hausman_k

2 years

DALL-E 2 🤯 "Teddy bears working on new AI research on the moon in the 1980s" Deep learning continues to overfit...

Karol Hausman

@hausman_k

4 years

It's just overfitting, they said. It just memorizes the data, they said.

23

184

2K

10

66

619

Karol Hausman

@hausman_k

1 year

PaLM-E or GPT-4 can speak in many languages and understand images. What if they could speak robot actions? Introducing RT-2: our new model that uses a VLM (up to 55B params) backbone and fine-tunes it to directly output robot actions!

19

117

600

Karol Hausman

@hausman_k

3 years

Super excited to announce that I've started as an Adjunct Professor @Stanford ! I'll continue to work @GoogleAI but I'll also be spending some time at Stanford, where I'll be co-advising a few students and continue co-teaching CS 330 () 🧑‍🏫

20

18

606

Karol Hausman

@hausman_k

2 years

Here are a few examples from our work in robotics that leverage bitter lesson 2.0 This is something that I believe we'll see a lot more of (including our own work in 2023)🧵

15

106

563

Karol Hausman

@hausman_k

2 years

⚠️🇺🇦🇺🇦🇺🇦🇵🇱🇵🇱🇵🇱⚠️ If you have friends/family trying to escape Ukraine and are looking for shelter in Poland, please DM me. My family and I can provide transport from the PL/RO border to my hometown (Koszalin) and shelter (+food/school/work etc) for as long as needed.

8

59

461

Karol Hausman

@hausman_k

2 years

Our 2021 CS330 () lectures are online: It was a pleasure to co-teach this class with @chelseabfinn . Topics incl. meta-learning, MTL, few-shot learning, deep RL (incl. multi-task, meta, goal-conditioned, hierarchical and offline RL)

Stanford CS330: Deep Multi-Task & Meta Learning I Autumn 2021I Professor Chelsea Finn

While deep learning has achieved remarkable success in supervised and reinforcement learning problems, such as image classification, speech recognition, and ...

www.youtube.com

8

80

441

Karol Hausman

@hausman_k

1 year

Our most recent work showing bitter lesson 2.0 in action: using diffusion models to augment robot data. Introducing ROSIE: Our robots can imagine new environments, objects and backgrounds! 🧵

10

58

416

Karol Hausman

@hausman_k

2 years

Bitter lesson by @RichardSSutton is one of the most insightful essays on AI development of the last decades. Recently, given our progress in robotics, I’ve been trying to predict what the next bitter lesson will be in robotics and how can we prevent it today. Let me explain 🧵

11

46

370

Karol Hausman

@hausman_k

4 years

Immigrants in the US are particularly vulnerable to #COVID19 layoffs. If you're on an H1-B work visa and you get laid off you need to find another job within 60 days. Almost nobody will be hiring for the next 60 days. We need a change of this policy ASAP!

7

50

321

Karol Hausman

@hausman_k

1 year

I've been in robotics and AI for 10+ years and this is the first time when it feels like there is at least the light at the end of the tunnel (for robotics, for AI it's more like we're staring at the sun)

12

20

262

Karol Hausman

@hausman_k

2 years

We have some exciting updates to SayCan! Together with the updated paper, we're adding new resources to learn more about this work: Interactive site: Blog posts: and Video:

Grounding language in robotic affordances

Welcome to ResearchBytes - a series that converts research publications into byte-sized content. In this first episode, we share how Google Research and Ever...

www.youtube.com

Karol Hausman

@hausman_k

2 years

Super excited to introduce SayCan (): 1st publication of a large effort we've been working on for 1+ years Robots ground large language models in reality by acting as their eyes and hands while LLMs help robots execute long, abstract language instructions

19

287

1K

3

57

246

Karol Hausman

@hausman_k

11 months

AI drone racer finally beats human world champions! Recipe: • state estimation with KF + gate-based measurements as additional KF updates • small deep RL policy trained in sim • residual controllers on top Nature article:

Davide Scaramuzza

@davsca1

11 months

We are thrilled to share our groundbreaking paper published today in @Nature : "Champion-Level Drone Racing using Deep Reinforcement Learning." We introduce "Swift," the first autonomous vision-based drone that beat human world champions in several fair head-to-head races! PDF

41

340

1K

4

54

236

Karol Hausman

@hausman_k

1 year

If you want to understand why robotics is much harder than it seems, @ericjang11 pointed me once to this essay that does a pretty good job explaining it: Reality has a surprising amount of detail

4

39

229

Karol Hausman

@hausman_k

7 months

This video blows my mind. And it will only get better from here.

7

30

230

Karol Hausman

@hausman_k

1 year

Large language models are universal computers that operate on tokens. They happened to be trained on language because we had a lot of it, but don't get it twisted - once trained, they are universal computers.

18

32

218

Karol Hausman

@hausman_k

10 months

🚨 New (Offline) RL Method 🦾 🚨 Introducing Q-Transformer - new RL approach that works at scale with large models and many tasks. This is the best method we found so far that works with demos and autonomous (also negative) data at large scale. 🧵

Yevgen Chebotar

@YevgenChebotar

10 months

Offline RL strikes back! In our new Q-Transformer paper, we introduce a scalable framework for offline reinforcement learning using Transformers and autoregressive Q-Learning to learn from mixed-quality datasets! Website and paper: 🧵

8

111

540

3

52

224

Karol Hausman

@hausman_k

10 months

ImageNet moment for robotics is finally here 🚀🦾 Introducing RT-X model & Open X-Embodiment dataset: Truly wonderful robotics community effort by 34 labs (173 authors!) led by @QuanVng showing that a general cross-embodied robotic brain is possible 🧵

Quan Vuong

@QuanVng

10 months

RT-X: generalist AI models lead to 50% improvement over RT-1 and 3x improvement over RT-2, our previous best models. 🔥🥳🧵 Project website:

7

144

622

6

40

209

Karol Hausman

@hausman_k

1 year

Lastly, another work from our lab @GoogleAI , this time without any foundation models: just super fast, agile, state-of-the-art control using modern and classical methods 🦾 3. Agile catching:

2

32

205

Karol Hausman

@hausman_k

4 years

One (somewhat forgotten) valuable aspect of RL is its ability to continuously improve. We present a simple fine-tuning method that shows this in challenging (often ridiculous) real-world scenarios. Paper: Website: Thread 👇

5

45

195

Karol Hausman

@hausman_k

7 months

I heard a wonderful analogy for AI progress from @demishassabis : It's hard to imagine the industrial revolution if we didn't happen to have dinosaurs under ground that we could use as fossil fuel. Without them, we'd need to wait to discover solar or nuclear. It's hard to

10

25

188

Karol Hausman

@hausman_k

5 months

In PIVOT we show how you can use VLMs without any fine-tuning for a wide range of control tasks - it's pretty crazy. We're just at the beginning of discovering what VLMs can do for control. website: demo:

Brian Ichter

@brian_ichter

5 months

How do you get zero-shot robot control from VLMs? Introducing Prompting with Iterative Visual Optimization, or PIVOT! It casts spatial reasoning tasks as VQA by visually annotating images, which VLMs can understand and answer. Project website:

5

22

103

3

36

186

Karol Hausman

@hausman_k

1 year

Growing up I had no idea that one can work on intelligent robots. Even in my undergrad (in robotics!), it seemed like all of robotics was about industrial automation. It all changed one day when I was looking for a bathroom.

7

14

185

Karol Hausman

@hausman_k

1 year

There are many reasons to work on embodied AI. One of them is that active data collection seems to allow for much better learning than passive observation, even if both active and passive agents are exposed to the same stimuli. Here are a few famous experiments on that 🧵

7

42

180

Karol Hausman

@hausman_k

2 years

We put everything together to see what RT-1 can enable in combination with SayCan. To make this more challenging, we evaluate RT-1 in two kitchens. We get 67% in Kitchen1 and the same perf in unseen Kitchen2, both significantly better than the baselines. Example execution:

5

54

175

Karol Hausman

@hausman_k

7 months

Robotics isn't easy even in the era of foundation models - you have to truly love it and be very committed, but it is possible. Having said that, it feels like for the first time in my career I see a path to making robotics work. 🦾

Dwarkesh Patel

@dwarkesh_sp

7 months

. @ilyasut on why OpenAI gave up on robotics: "Progress comes from the combination of compute and data. There was no path to data from robotics. I'd say that now it is possible to create a path forward, but one needs to really commit. You could imagine a gradual path of

51

92

979

5

9

171

Karol Hausman

@hausman_k

1 year

@chelseabfinn and I will ll be teaching a new class on Deep RL @Stanford this upcoming quarter: CS224R: Yup, we'll post all the videos after 🦾 🎓

3

18

168

Karol Hausman

@hausman_k

2 years

Training RL from scratch can be very hard, if there is any prior policy you can use to help, you should. But using prior policies with value-based RL is difficult for various RL reasons. Introducing Jump-Start RL: a simple, widely applicable method that addresses this problem.

Google AI

@GoogleAI

2 years

A key challenge in #ReinforcementLearning is learning policies from scratch in environments with complex tasks. Read how a meta-algorithm, Jump Start Reinforcement Learning, uses prior policies to create a learning curriculum that improves performance →

5

93

291

5

36

164

Karol Hausman

@hausman_k

2 years

1) We updated the underlying LLM to PaLM (), resulting in PaLM-SayCan. This resulted in an interesting trend: Improving the underlying LLM resulted in much higher robotics (!) performance (halving the errors)

4

26

161

Karol Hausman

@hausman_k

3 years

There is a new format for @GoogleAI (Brain) internships: 1) If you are in your final year, apply for an internship: 2) If you are at the beginning of your PhD, apply for a student researcher position:

0

39

160

Karol Hausman

@hausman_k

6 months

Not many people realize (or share the belief) that with enough data, robotics will be solved. If that's case, we simplified a very hard problem into a much easier problem: robotics -> how to get a lot of robot data?

32

10

144

Karol Hausman

@hausman_k

2 years

Super excited to announce this year’s Deep RL workshop @NeurIPSConf 🎉🎉🎓🎓 website: submission deadline: September 22nd, 11:59 PM PST Some exciting changes that we made this year in 🧵👇

1

26

155

Karol Hausman

@hausman_k

6 months

No matter how complex or simple your robotics system is, there's always a bug in the rotation matrix. Always.

Peter Mitrano

@PeterMitrano

6 months

I think there might be a lot of imposter syndrome specifically around 3D rotations and coordinate transforms. So let it be known that I have and continue to struggle debugging these issues! Is it actually hard for everyone! It's ok if it feels hard!

13

4

118

10

8

151

Karol Hausman

@hausman_k

10 months

Agility has built a humanoid robot factory that should be able to manufacture up to 10k humanoid robots a year. 🤖 🦾 Announcement here:

5

20

150

Karol Hausman

@hausman_k

10 months

For folks asking: yes, we fully open-sourced the RT-1/RT-2 dataset as part of Not only you can train on that data now but you can also merge it with 60 other datasets that are all in the same format with colabs etc. showing how to do it.

1

28

147

Karol Hausman

@hausman_k

2 years

Based on the success of large models in other fields, our goal was to build a model that acts as an “absorbent data sponge”. It should get better with diverse, multi-task data. The more data the model can effectively absorb, the better it will be able to generalize.

2

12

136

Karol Hausman

@hausman_k

7 months

RL has many different gifts and it's easy to get confused which ones are the most important, especially in the age of gigantic BC-based models. Here is a short, non-exaustive list: • mining hard negatives • stitching (getting more out of the same data) • optimizing an

5

15

137

Karol Hausman

@hausman_k

1 year

5 reasons why you should check out PaLM-E: 🌴 🤖 Some of these results were pretty mind-blowing to us! 🤯 🧵 👇

Karol Hausman

@hausman_k

1 year

I sure talk a lot about foundation models for robotics and how we can ride their wave 🌊 but this time we've trained our own: We present PaLM-E: An Embodied Multimodal Language Model Danny has some incredible results in 🧵👇 so hold on tight!

1

16

97

6

21

137

Karol Hausman

@hausman_k

2 years

Very cool work from my colleagues at Robotics @ Google If you want to have multimodal models, you don't have to train them, just have individual foundational models talk to each other Who would've thought that best language for AI to communicate with itself will be our language

Andy Zeng

@andyzeng_

2 years

With multiple foundation models “talking to each other”, we can combine commonsense across domains, to do multimodal tasks like zero-shot video Q&A or image captioning, no finetuning needed. Socratic Models: website + code: paper:

21

382

2K

4

19

134

Karol Hausman

@hausman_k

10 months

Robotics is becoming the next frontier of AI resaerch. While there are many companies being built on top existing LLM APIs and there is another wave of virtual agents companies coming, robotics remains in the sweet spot of still having many open problems & huge impact to be had.

4

6

134

Karol Hausman

@hausman_k

8 months

How to know if a new scientific trend is legit? Professors are skeptical, while students can't stop talking about it. Students, not professors, set trends. Max Planck put it in much more morbid terms: A new scientific truth does not triumph by convincing its opponents and

9

11

134

Karol Hausman

@hausman_k

7 months

It's been a huge year for robotics and foundation models 🦾 🎉 Here's a thread showing some important (though def biased towards my team) works that happened in 2023:

1

20

135

Karol Hausman

@hausman_k

10 months

Summary of robot table tennis work from our team: This is a fairly learning complex system with multiple components and many lessons learned. I encourage you to take a look at the website for details.

5

26

135

Karol Hausman

@hausman_k

7 months

I believe that the biggest application of the *multimodal* models will be robotics and embodied agents. We're actively working on making this future happen. @demishassabis 's quote in the Wired article on Gemini by @willknight below:

4

6

129

Karol Hausman

@hausman_k

4 years

Specifying rewards in real-world RL is hard - unsupervised RL can change that! We present Off-DADS, an off-policy version of our DADS algorithm that is sample-efficient enough to allow experiments on real robots. Arxiv Video: 1/3

1

31

125

Karol Hausman

@hausman_k

1 year

The next work from our lab @GoogleAI and collaborators is on using LLMs for gait generation for quadrupeds: 2. Language to locomotion: How would you specify low-level control commands to a quadruped? 👇

1

26

121

Karol Hausman

@hausman_k

11 months

Many researchers have asked us about sharing our RT dataset and making it easier to participate in large-scale robot learning research. We're working on it and we'll have some updates on this soon! 👀

6

11

124

Karol Hausman

@hausman_k

2 years

As for the model - yup, it's a Transformer😊We tokenize images using early language fusion with FiLM layers added to ImageNet-pretrained EfficientNet. We use TokenLearner to limit # of tokens (15ms total inference time) and output action tokens for the arm and base actions.

1

13

118

Karol Hausman

@hausman_k

1 year

It's much nicer to look at research as a collaborative endeavor against the problem, rather than a competitive endeavor against other researchers.

2

13

121

Karol Hausman

@hausman_k

1 year

RT-2 is available on arxiv now: Here is a video showing some of its capabilities:

6

31

119

Karol Hausman

@hausman_k

2 years

@OpenAI demo on interactive Codex that debug its own code and make corrections on the fly. Coding is changing before our eyes🤯 We'll post some exciting updates on robotics applications of this tech in a few days, stay tuned!

Satya Nadella & Sam Altman Conversation (Satya Nadella at Microsoft...

Audio Described Version: https://youtu.be/ekNwRWwPCrcSubscribe to Microsoft on YouTube here: https://aka.ms/SubscribeToYouTubeFollow us on social: LinkedIn: ...

www.youtube.com

5

18

119

Karol Hausman

@hausman_k

2 years

Speaking of data, we collected over 130k demos (17+ months with 13 robots). The current set of skills includes picking, placing, opening/closing drawers, getting items in and out drawers, placing elongated items up-right, knocking them over, pulling napkins and opening jars.

4

13

108

Karol Hausman

@hausman_k

1 year

I've had a few students ask me whether it makes sense to work on LLMs in robotics and robot learning methods or is Google gonna solve it all. I can't emphasize it enough - you definitely should! This is a very new space that you can explore way more efficiently than Google can.

2

5

112

Karol Hausman

@hausman_k

5 years

Quite an emotional roller coaster with #NeurIPS2019 : our initial submission had scores 9,7,6 which after rebuttal increased to 9,8,7. The paper eventually got rejected due to the intervention of the meta-reviewer who spotted a mistake in the derivation...

6

113

Karol Hausman

@hausman_k

1 year

I usually talk about how robotics should ride the wave of foundation models, but yesterday I was told that another wave is coming: cheaper and more reliable robot hardware from Chinese manufacturers. I hope that they could do for bi-arm robots what Unitree did for quadrupeds.

11

4

109

Karol Hausman

@hausman_k

1 year

Quick update: Code as Policies won the Outstanding Learning Paper Award at #ICRA2023 !

Jacky Liang @ RSS 2024

@jackyliang42

1 year

Looking forward to presenting Code as Policies this week #ICRA2023 @ieee_ras_icra ! Talk: Award Finalists 2, Tuesday 4PM Poster: Thursday 9-10:40AM Come see how code-writing LLMs enable robots to do novel & diverse tasks from language instructions!

0

6

43

4

10

110

Karol Hausman

@hausman_k

6 months

AutoRT is here: foundation models 🤝 robots at scale We show how to use VLMs and LLMs to orchestrate a fleet of robots and allow 1:5 human:robot ratio 💪🦾 Led by @keerthanpg and @AlexIrpan Fun fact: you can now literally write Asimov's laws into the

Keerthana Gopalakrishnan

@keerthanpg

6 months

In the last two years, large foundation models have proven capable of perceiving and reasoning about the world around us unlocking a key possibility for scaling robotics. We introduce a AutoRT, a framework for orchestrating robotic agents in the wild using foundation models!

18

152

821

0

16

107

Karol Hausman

@hausman_k

2 years

@keerthanpg @xiao_ted @xf1280 @julianibarz @YevgenChebotar @AlexIrpan @ryancjulian @svlevine @chelseabfinn @pannag_ @IMordatch @ofirnachum @KarlPertsch @TianheYu You can find more details on RT-1 below. Paper: (soon on arxiv) Website: Video: Blog: More high-level blog:

Helping robots learn from each other

A look at how Transformers are helping robots become more useful.

blog.google

2

15

105

Karol Hausman

@hausman_k

1 year

Recent articles about RL such as and touch upon a concept that we found to be crucial to consider in robot learning: Data sponge 🧽 🧵

2

17

104

Karol Hausman

@hausman_k

11 months

"This rebuttal resolved all of my concerns and answered all the remaining questions. In fact, this is an exemplary piece of work that should be posted on every forum and be taught in every grad school. This paper is truly perfect now, thank you! I maintain my score. Weak accept."

3

0

100

Karol Hausman

@hausman_k

1 year

"The best way to understand intelligence is to build it" didn't turn out as we thought. The more we build it, the less we understand how it works.

10

96

Karol Hausman

@hausman_k

1 year

🚨 🚨 Another new work showcasing bitter lesson 2.0 🚨 🚨 Introducing MOO: We leverage vision-language models (VLMs) to allow robots to manipulate objects they've never interacted with, and in new environments, while learning end-to-end policies. 🧵

3

20

98

Karol Hausman

@hausman_k

1 year

I sure talk a lot about foundation models for robotics and how we can ride their wave 🌊 but this time we've trained our own: We present PaLM-E: An Embodied Multimodal Language Model Danny has some incredible results in 🧵👇 so hold on tight!

Danny Driess

@DannyDriess

1 year

What happens when we train the largest vision-language model and add in robot experiences? The result is PaLM-E 🌴🤖, a 562-billion parameter, general-purpose, embodied visual-language generalist - across robotics, vision, and language. Website:

32

523

2K

1

16

97

Karol Hausman

@hausman_k

3 years

#MarsHelicopter is on the surface of Mars! Here is a picture of our a little drone prototype that we were using @NASAJPL to test our state estimation algorithms 5 yrs ago. It's come a long way!

1

5

98

Karol Hausman

@hausman_k

3 years

We ( @narayan_avnish , @hayden_shively , @adibellathur , @ryancjulian , @TianheYu , @svlevine , @chelseabfinn and myself)'ve been working hard over the past year to improve the Meta-World benchmark (). Here are a few exciting updates:🧵

1

11

94

Karol Hausman

@hausman_k

2 years

SayCan is available in Roblox now! You can play with an interactive agent using supported by GPT-3

First look - GPT-3 in Roblox - Emulating the functionality of Google...

Play now: https://www.roblox.com/games/11462889413/GPT-3-SayCan-NPCJames: https://github.com/JavaFXpertBased on the CoT paper: https://arxiv.org/abs/2201.119...

www.youtube.com

4

28

90

Karol Hausman

@hausman_k

1 year

Our new work on Grounded Decoding: shows how multiple models can participate in the LLM decoding process. You can now ground your LLM token by token as long as you have a grounding model.

1

13

90

Karol Hausman

@hausman_k

4 years

Imagine being an international student, you finished all your courses & you're getting close to graduation: 1. Your OPT (gives you a temp. work permit) will likely be cancelled. 2. H1B can be cancelled anytime too. 3. You can't really go home because of the pandemic. And now:

Andrew Ng

@AndrewYNg

4 years

New @ICEgov policy regarding F-1 visa international students is horrible & will hurt the US, students, and universities. Pushes universities to offer in-person classes even if unsafe or no pedagogical benefit, or students to leave US amidst pandemic and risk inability to return.

49

836

4K

2

12

91

Karol Hausman

@hausman_k

2 years

We're releasing one of the biggest real-world multi-task RL robotic datasets! Our MT-Opt dataset has ~ 1M (!) RL trajectories and ~0.5M images used to train the MT-Opt Q function and the success detector. See specs and more details:

mt_opt | TensorFlow Datasets

www.tensorflow.org

1

25

91

Karol Hausman

@hausman_k

8 months

The reason why physical AI is harder than generative AI is because of their respective customers. Humans can easily forgive any errors, physics is unforgivable.

7

2

89

Karol Hausman

@hausman_k

1 year

Do you believe that robotics could be solved if someone spent 1-2 billion dollars on it? If so, how would you solve it with this kind of money?

60

3

89

Karol Hausman

@hausman_k

11 months

It seems like a recipe for a robot specialist is fairly clear: • state estimation if you can • deep RL with sim2real • fine-tuning in real Could be also done fully in real if you can collect the data. It seems very different from robot generalist recipes of today.

6

91

Karol Hausman

@hausman_k

3 months

My account was hacked over the weekend but it's secure now. If you ever see me posting about crypto, it's either my account got hacked and you should contact me immediately or I need an intervention and you should contact me immediately

19

2

84

Karol Hausman

@hausman_k

2 years

In SayCan, we showed how we can connect robot learning pipelines to large language models, bringing a lot of common sense knowledge to robotics. The hope was that as the LLMs become better (which they seem to be consistently doing), it will have a positive effect on robotics.

Karol Hausman

@hausman_k

2 years

Super excited to introduce SayCan (): 1st publication of a large effort we've been working on for 1+ years Robots ground large language models in reality by acting as their eyes and hands while LLMs help robots execute long, abstract language instructions

19

287

1K

2

4

87

Karol Hausman

@hausman_k

1 year

It's kind of weird to fully internalize that AGI will happen. The timeline isn't clear but at this point it's pretty clear that it will happen.

9

7

83

Karol Hausman

@hausman_k

8 months

Many people are talking about a ChatGPT moment in robotics. Before we achieve a ChatGPT moment, we need the Internet moment first. I hope that RT-X can contribute to that:

2

8

86

Karol Hausman

@hausman_k

1 year

I often hear: why work on embodied AI given how much easier it has become to work on {digital agents, language, vision, LLM startups}?

5

11

86

Karol Hausman

@hausman_k

1 year

Maybe the fact that LLMs appear intelligent says more about us than it does about AI. Many of the words we say just come from statistics of what word comes after another and not from a conscious thought. It's surprisingly difficult to say something that breaks the statistics.

5

11

82

Karol Hausman

@hausman_k

6 years

I'm happy to announce that I'll be starting @GoogleBrain in Mountain View tomorrow as a Research Scientist!

1

85

Karol Hausman

@hausman_k

3 years

Imitation learning (IL) is good at getting baseline performance from demos quickly but it has trouble improving. RL takes a long time initially but it can continuously improve. In AW-Opt (), we develop an IL+RL algorithm to combine the benefits of both.🧵

1

9

84

Karol Hausman

@hausman_k

5 years

Can we learn meaningful behaviors and their dynamics without any rewards? Yes! And we can solve new tasks zero-shot by using MPC to compose the learned skills. Dynamics-Aware Discovery of Skills (DADS): w/ @archit_sharma97 , Shane Gu, @svlevine , @Vikashplus

1

12

82

Karol Hausman

@hausman_k

2 years

Basic idea: With prompt engineering and scoring we can use LLM to break down an instruction into small, actionable steps. This is not enough though, the LLM doesn't know about the scene, embodiment and the situation it's in. It needs what we call an affordance function!

1

4

80

Karol Hausman

@hausman_k

2 years

@keerthanpg @xiao_ted @xf1280 @julianibarz @YevgenChebotar @AlexIrpan @ryancjulian @svlevine @chelseabfinn @pannag_ @IMordatch @ofirnachum @KarlPertsch @TianheYu Ah, two more team members are on Twitter: @SOTA_kke and @kuanghueilee Also, here is the code:

GitHub - google-research/robotics_transformer

Contribute to google-research/robotics_transformer development by creating an account on GitHub.

github.com

7

11

79

Karol Hausman

@hausman_k

2 years

Deep RL workshop @NeurIPSConf starts on Friday! Hear invited talks from @tobigerstenberg on counterfactual simulation of causal judgments, @j_foerst on opponent-shaping in games, @IMordatch on sequence modeling, @yayitsamyzhang on learning generalist agents. Don't miss out! 🧵

1

20

81

Karol Hausman

@hausman_k

9 months

Every single time I'm stuck on a coding problem, I have to repeat to myself that I should just try parts of it in a colab to understand it. Action drives understanding way more than passive thinking does

2

3

80

Karol Hausman

@hausman_k

1 year

There has been a long debate on model-based vs model-free RL. The classic arguments include: • MBRL has richer learning signal • MBRL is task-independent • MFRL optimizes what you care about • MFRL is data inefficient I don't think this dichotomy is quite right. 🧵

5

7

81

Karol Hausman

@hausman_k

1 year

The version of RLHF that is used rn is rumored to be a weak RL algo (more of a filtered BC) that optimizes for a human-based reward model. The next versions will likely use more powerful RL + optimize for more long-horizon rewards like a conversation outcome.

8

7

78

Karol Hausman

@hausman_k

3 years

Resets are one of the most limiting, often under-emphasized requirements of current robotic RL methods. They are hard to automate and scale to multiple tasks. We introduce Value-accelerated Persistent Reinforcement Learning (VaPRL) that tries to address this problem.

1

11

77

Karol Hausman

@hausman_k

2 years

Remember the kitchen env from Relay Policy Learning? This time it's in real! In DBAP, we create system that continuously, autonomously improves on many tasks. It's not enough to give demos to bootstrap tasks, demos should also bootstrap practicing! 🧵👇

1

8

77

Karol Hausman

@hausman_k

1 year

We'll have some very exciting new work to share on Friday this week 🤖 Stay tuned! 👀 👀

5

2

78