Dhruv Batra @DhruvBatraDB Twitter profile | Pikagi

Pikagi

Dhruv Batra

@DhruvBatraDB

15,778

Followers

397

Following

238

Media

2,379

Statuses

AI Researcher. Professor ( @GeorgiaTech ). Prev: Senior Director leading FAIR Embodied AI ( @MetaAI ). Co-founded CaliperAI. @CarnegieMellon alum.

https://t.co/ptkUFEK5QZ

Joined January 2016

Don't wanna be here? Send us removal request.

Pinned Tweet

@DhruvBatraDB

Dhruv Batra

2 months

Update: After nearly 8 years, I have left Meta. I was leading FAIR’s Embodied AI team and I did the best work of my life here. My colleagues and I imagined a world where AI agents can see, talk, and act. And we pulled that future closer by building the required pieces –

71

29

868

Last Seen Profiles

@EnnsZoneKicking

@VmaniakJ

@napalm2050

@melodores

@Havent11197

@kamnaqueen

@broncobellx

@zeesaias22

@mym11tenki_trhk

@JSolz

@BrodheadSh61848

@Maxairjordan1

@wiarytt

@feelthebyrn1

@YoNoNunca

@diditakeit2farr

@stw_pdg

@VeroLaflamme

@3sUF5YuPYwJmZ94

@PopOp378272

@hirayoshir74246

@AST_TYFCOP

@UwUrch1337

@felizMIDORI

@mred789031812

@KasihLudah

@nignizhuo

@MLUIntl

@B00m3r09

@WowStation

@MomoMomo573105

@BrujulaNoticias

@BigJmost45

@follow87653932

@dr_korayates

@SeanMoore07

@DhruvBatraDB

Dhruv Batra

2 years

We are entering a new phase in generative models. Text-to-video is here! Make-a-video by @MetaAI and FAIR. Look at this video! It's generated! "A golden retriever eating ice cream on a beautiful tropical beach at sunset, high resolution"

17

234

1K

@DhruvBatraDB

Dhruv Batra

1 year

Every branch of science has its corresponding pseudoscience. Astronomy has astrology. Geophysics has flat-earth beliefs. Chemistry had (has?) alchemy. Evolutionary biology has creationism. AI now has AGI existential risk. Maybe it’s a sign of maturing as a field.

128

105

923

@DhruvBatraDB

Dhruv Batra

2 years

A thought-experiment to inspire scientists is to ask: If you could write only 20 papers in your lifetime, would your current work be one of them? This is one of my 20. 🧵👇

Tweet card media

Emergence of Maps in the Memories of Blind Navigation Agents

Animal navigation research posits that organisms build and maintain internal spatial representations, or maps, of their environment. We ask if machines -- specifically, artificial intelligence...

15

128

857

@DhruvBatraDB

Dhruv Batra

1 year

Contemporary discussion (hype?) about LLMs and “pausing AGI development” seems oblivious of Moravec’s paradox. We’ve hypothesized since the 80s — that the hardest problems in AI involve sensorimotor control, not abstract thought or reasoning. It

22

152

806

@DhruvBatraDB

Dhruv Batra

4 months

I have been working on vision+language models (VLMs) for a decade. And every few years, this community re-discovers the same lesson -- that on difficult tasks, VLMs regress to being nearly blind! Visual content provides minor improvement to a VLM over an LLM, even when these

Tweet media one

@AIatMeta

AI at Meta

4 months

Today we’re releasing OpenEQA — the Open-Vocabulary Embodied Question Answering Benchmark. It measures an AI agent’s understanding of physical environments by probing it with open vocabulary questions like “Where did I leave my badge?” More details ➡️

38

259

1K

23

115

780

@DhruvBatraDB

Dhruv Batra

10 months

Announcing Habitat 3.0, simulating humanoid avatars and robots collaborating! - Humanoid sim: diverse skinned avatars - Human-in-the-loop control: mouse/keyboard or VR - Tasks: social navigation and rearrangement Over 1,000 steps per second on 1 GPU for large-scale learning!

9

88

494

@DhruvBatraDB

Dhruv Batra

3 years

I am excited to announce Season 2 of Humans of AI: Stories, Not Stats! @deviparikh created this series in 2020 and I am the host for Season 2, where I interview a cohort of 20 AI researchers to learn more about them.

Tweet media one

8

49

367

@DhruvBatraDB

Dhruv Batra

5 years

Today, @facebookai and @mlatgt ( @gtcomputing ) announced that they will be partnering to co-teach an AI class (CS 4803/7643 Deep Learning) to a diverse body of students at @GeorgiaTech . This is an innovative model, and I'm proud of the hard work of my colleagues on both sides.

Tweet media one

Tweet media two

Tweet media three

Tweet media four

7

55

331

@DhruvBatraDB

Dhruv Batra

4 months

I don’t often subtweet, but when I do, it is to say that I’m still at FAIR.

8

2

297

@DhruvBatraDB

Dhruv Batra

5 years

Excited to announce Habitat, a platform for embodied AI research: — Habitat-Sim: high-perf 3D sim (w/ SUNCG, MP3D, Gibson) — Habitat-API: modular library for defining tasks, training agents — Habitat-Chal: autonomous nav challenge on #EvalAI @facebookai

2

71

247

@DhruvBatraDB

Dhruv Batra

2 years

#CVPR2023 reviews are out & I am reminded again of how many reviewers don't understand their role. Our job is not to tell authors to write papers as we'd write them. Our job is to gauge correctness & significance. Whether a paper conforms to our writing style is irrelevant!

4

14

241

@DhruvBatraDB

Dhruv Batra

3 months

FAIR researchers ( @AIatMeta ) presented SegmentAnything and our robotics work at the White House correspondents’ weekend. Llama3 + Sim2Real skills (trained with @ai_habitat ) = a robot assistant

Tweet media one

Tweet media two

Tweet media three

Tweet media four

@thehill

The Hill

3 months

Washingtonians delved into the world of artificial intelligence (AI) at the Washington AI Network’s inaugural weekend TGAIFriday Lunch for White House correspondents.

21

7

67

3

19

224

@DhruvBatraDB

Dhruv Batra

5 years

I am grateful for the honor and recognition. I get to be the face of this, but there's a team (nay family) of students, post-docs, colleagues, and other collaborators that make this possible.

@mark_riedl

Mark Riedl

5 years

Congratulations to @DhruvBatraDB for winning a Presidential Early Career Award for Scientists and Engineers (PECASE) PECASE is the highest honor bestowed by the US government for early-career scientific research @GeorgiaTech had 3 winners this year

1

3

85

13

4

206

@DhruvBatraDB

Dhruv Batra

1 year

FAIR and ⁦ @gtcomputing ⁩ researchers demoing our sim2real work on ⁦ @BostonDynamics ⁩ Spot at ⁦ @MetaAI ⁩ booth at #CVPR23 .

5

27

169

@DhruvBatraDB

Dhruv Batra

4 years

Here's what we've been up to. Work done in collaboration between @facebookai , Facebook Reality Labs, Georgia Tech ( @gtcomputing , @mlatgt ), Oregon State, University of Illinois, and University of Texas at Austin.

1

34

156

@DhruvBatraDB

Dhruv Batra

3 years

HM3D paper is now out.

@_akhaliq

AK

3 years

Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AI pdf: abs:

Tweet media one

1

34

176

1

24

145

@DhruvBatraDB

Dhruv Batra

1 year

Last 4 years of @ai_habitat have been a steady march against moving goalposts: — Model-free RL will never scale: Yes, it does with a fast sim, DD-PPO and VER — Performance in sim will never generalize to robots: Yes, it does and

Tweet card media

VER: Scaling On-Policy RL Leads to the Emergence of Navigation in...

We present Variable Experience Rollout (VER), a technique for efficiently scaling batched on-policy reinforcement learning in heterogenous environments (where different environments take vastly...

@EugeneVinitsky

Eugene Vinitsky

@EugeneVinitsky

1 year

Paper after paper from the Habitat folks is showing that fast, physics-free simulation + distributed PPO is a superbly scalable strategy. Amazing work.

0

6

52

2

17

132

@DhruvBatraDB

Dhruv Batra

5 years

Yeah, no. We can *barely* simulate photo-realistic rendering. Physics much less so (see e.g. @ai_habitat ). "Simulating" language is much harder still. "Every conceivable social interaction"? Nope.

@newscientist

New Scientist

5 years

Predicting the future is now possible with powerful new AI simulations that can model every conceivable social interaction

Tweet media one

168

107

347

2

18

131

@DhruvBatraDB

Dhruv Batra

3 years

This is what we’ve been up to for the last 2 years. Proud of the effort by a large interdisciplinary team.

@ai_habitat

AI Habitat

3 years

Two major releases today: 1. Habitat-Matterport 3D dataset: largest-ever public dataset of 1,000 3D scans of indoor spaces. 2. Habitat 2.0: our next generation simulator for training mobile manipulation robots. Sim-speed over 25k steps/ sec (850× real-time) on an 8-GPU node

Tweet media one

Tweet media two

3

29

133

3

4

129

@DhruvBatraDB

Dhruv Batra

2 months

Yes, this didn’t age well. It was true when I wrote it. Don’t @ me bro.

@DhruvBatraDB

Dhruv Batra

4 months

I don’t often subtweet, but when I do, it is to say that I’m still at FAIR.

8

2

297

2

1

125

@DhruvBatraDB

Dhruv Batra

9 months

FAIR Embodied AI team is hiring research interns. Cutting-edge work in robotics, AR/MR, sim2real transfer, egocentric CV, pre-training for embodied agents — all in an open fundamental research environment.

Tweet card media

Meta's mission is to give people the power to build community and bring the world closer together. Together, we can help people build stronger communities - join us.

www.metacareers.com

0

20

121

@DhruvBatraDB

Dhruv Batra

5 years

. @deviparikh featured in Vogue's Women in AI article!

@mlatgt

Machine Learning at Georgia Tech

5 years

"I think there is a huge gap in terms of what technology can do today versus what people think it can do." - @deviparikh , in @voguemagazine 's "Women in AI" article. 🖥️:

Tweet media one

0

8

51

0

14

122

@DhruvBatraDB

Dhruv Batra

7 years

Excited to announce Embodied Question Answering: — An agent is spawned in a 3D environment and asked a question (‘What color is the car?’). — It must intelligently navigate the environment and gather information via first-person vision to answer the question (‘orange’).

2

40

121

@DhruvBatraDB

Dhruv Batra

8 months

Surely it is a market failure that American cities (like SF) don’t have more chai shops. Not the abomination sold as chai (tea) latte, actual desi boil-the-tannins-out chai. The demand is there. Why isn’t there more supply?

8

3

115

@DhruvBatraDB

Dhruv Batra

3 years

S2E20 is out! Yejin Choi @YejinChoinka ( @uwcse , @allen_ai ) on Humans of AI: Stories, Not Stats. Yejin talks about where she comes from, living life like in a game environment, thinking in vector spaces, finding her true self, and lots more. [1/n]

Tweet media one

1

14

119

@DhruvBatraDB

Dhruv Batra

5 years

Physics: coming soon to a simulator near you.

@ai_habitat

AI Habitat

5 years

Here's an early peek into a major new functionality coming in Habitat -- importing objects and simulating physics (push, pull, poke). Follow the progress on this massive WIP PR:

0

14

70

2

18

113

@DhruvBatraDB

Dhruv Batra

1 year

First, we have developed an artificial visual cortex (called VC-1) for embodied AI. A single perception model that supports a diverse range of sensorimotor skills, environments, and embodiments. VC-1 matches or outperforms best-known results on 17 different sensorimotor tasks!

2

14

111

@DhruvBatraDB

Dhruv Batra

5 years

. @JeffDean speaking at @mlatgt about Deep Learning at @GoogleAI .

Tweet media one

Tweet media two

Tweet media three

Tweet media four

1

4

111

@DhruvBatraDB

Dhruv Batra

3 months

. @eccvconf reviews are out and the official notification email points to a blog @deviparikh , @stefmlee , and I wrote a few years ago for our students. Glad to see that our lab style is spreading to the community :-).

Tweet card media

How we write rebuttals

By Devi Parikh, Dhruv Batra, Stefan Lee

deviparikh.medium.com

5

12

107

@DhruvBatraDB

Dhruv Batra

1 year

Getting ready for the demo at #CVPR23 @MetaAI booth tomorrow.

1

12

104

@DhruvBatraDB

Dhruv Batra

1 year

Second, Adaptive Skill Coordination (ASC) for long-horizon tasks like tidying a house. ASC deployed on @BostonDynamics Spot achieves near-perfect performance on mobile pick-and-place — navigating to a counter, finding an object, picking it, navigating, placing, repeating.

2

16

97

@DhruvBatraDB

Dhruv Batra

5 years

What she said 👇. Couldn’t have said it better.

@deviparikh

Devi Parikh

5 years

. @DhruvBatraDB and I got tenure! Thank you @ICatGT @gtcomputing @mlatgt . Most of all, thanks to our students, postdocs and research scientists in the CVMLP labs -- first at Virginia Tech, and now at Georgia Tech -- for all the wonderful work over the years! You make this home.

Tweet media one

29

14

383

4

1

97

@DhruvBatraDB

Dhruv Batra

2 years

@ylecun @shelan The number matters a lot in developing nations. USD 100 / paper would cause significant reduction in submissions from such nations. Not because their idea are lacking, but because they won’t be able to risk it. Not quite the incentive we want.

2

2

94

@DhruvBatraDB

Dhruv Batra

3 years

13 challenges. 6 workshop organizers. 42 challenge organizers. 17 scientific advisors. 21 organizations. 1 workshop: #CVPR2021

@AIatMeta

AI at Meta

3 years

We're excited to launch the third Habitat Challenge at the Embodied AI workshop with 15 research & academic institutions. The 2021 Habitat Challenge invites AI experts from around the world to teach machines to navigate real-world environments. #CVPR2021

6

65

260

0

16

91

@DhruvBatraDB

Dhruv Batra

4 years

Congratulations @aagrawalAA ! Aishwarya’s thesis made fundamental contributions to the sub-field of vision+language through her work on VQA, and helped create a vibrant community. UdeM/MILA just got another AI expert :-).

@mlatgt

Machine Learning at Georgia Tech

4 years

Congratulations to ML @GT alumna Aishwarya Agrawal on being named a runner-up for the 2019 AAAI/ACM SIGAI Dissertation Award 🎉 She will be honored at #AAAI2021 . Next month, Agrawal will start as an assistant professor at the University of Montreal and Mila.

Tweet media one

0

1

79

1

1

88

@DhruvBatraDB

Dhruv Batra

5 years

Hey #NLP / Grounded Language folks -- VLN is now in Habitat. Instruction following ("Go outside the room, stop at the brown door") with continuous state-space navigation. Thanks @jacob__krantz and @Akakoshy !

@ai_habitat

AI Habitat

5 years

New in Habitat: Vision-and-Language Navigation. VLN task/dataset (by @panderson_me ) asks an agent to follow human nav-instructions in new buildings. Habitat Port Courtesy: @jacob__krantz @Akakoshy (students at @OregonState ), w/ @stefmlee @o_maksymets

1

20

65

0

20

85

@DhruvBatraDB

Dhruv Batra

6 years

Thank you! I'm honored, and fortunate to work with a fantastic group of students, post-docs, and collaborators.

@mlatgt

Machine Learning at Georgia Tech

6 years

Huge congratulations to #MLatGT faculty member @DhruvBatraDB on being named a recipient of the prestigious Early Career Award for Scientists and Engineers (ECASE-Army) by the Army Research Office. We're so proud of you! 🏆:

Tweet media one

3

2

59

3

3

81

@DhruvBatraDB

Dhruv Batra

1 year

A lot of my arguments about the foundations of intelligence being sensorimotor control (and not language or reasoning) are shaped by discussions with Jitendra over the years. This is a good summary of his arguments.

@JitendraMalikCV

Jitendra MALIK

@JitendraMalikCV

1 year

I delivered the 110th Annual Martin Meyerson UC Berkeley Faculty Research Lecture on March 20, 2023.

11

26

269

4

5

78

@DhruvBatraDB

Dhruv Batra

2 years

@thegautamkamath @CSrankings It is intellectually dishonest to call it CSRankings. It would be more intellectually honest to call it MyRankings. Because it is run by a single person’s rules. It is not a community project. We shouldn’t pretend otherwise.

3

1

79

@DhruvBatraDB

Dhruv Batra

3 months

From Best Paper Award finalist to winner! Congratulations @naokiyokoyama0 on a well-deserved recognition! @gtcomputing @GTrobotics #icra2024

Tweet media one

Tweet media two

Tweet media three

@DhruvBatraDB

Dhruv Batra

3 months

Naoki Yokoyama @naokiyokoyama0 presenting his best paper award finalist talk at #ICRA2024 ! Vision-Language Frontier Maps for Zero-Shot Semantic Navigation: show how to combine VL foundation models with a mapping+search stack. @ICatGT @GTrobotics @mlatgt @BostonDynamics

Tweet media one

Tweet media two

Tweet media three

Tweet media four

1

5

55

1

8

79

@DhruvBatraDB

Dhruv Batra

2 years

Come participate in the Habitat Rearrangement Challenge at @NeurIPSConf 2022.

@ai_habitat

AI Habitat

2 years

Announcing: Habitat Rearrangement Challenge at NeurIPS 2022! Goal: Control a home assistant robot and rearrange 1 object from start to goal position.

1

19

101

4

16

79

@DhruvBatraDB

Dhruv Batra

5 years

Excited to share this collaborative project by FAIR & FRL. If we can train a virtual bot to locate keys in a virtual home, a robot should eventually be able to do that in reality. Replica dataset provides the (hyper)realism. Habitat platform provides the (extremely fast) sim.

@AIatMeta

AI at Meta

5 years

We’re open sourcing AI Habitat, a powerful new simulation platform for training agents in hyperdetailed, photorealistic 3D reconstructions of physical environments. We hope this research milestone will unify & accelerate the promising field of Embodied AI.

8

144

450

2

14

78

@DhruvBatraDB

Dhruv Batra

3 years

Season 2 Episode 2 is out! Ray Mooney ( @UTCompSci ) on Humans of AI: Stories, Not Stats. Ray talks about how he finds joy in brainstorming ideas with his students, making an impact by doing what one loves, his fascination with the evolution of human intelligence and a lot more

Tweet media one

1

14

72

@DhruvBatraDB

Dhruv Batra

4 years

I was a mix of patient 0 and a test subject :-). More seriously, I am privileged to have early access. I learned a lot about these folks and am grateful they opened up and shared their stories.

@deviparikh

Devi Parikh

4 years

Very excited to introduce Humans of AI: Stories, Not Stats! In this series, I interview AI researchers to get to know them better as people. Starting next week, I will release two interviews every week as videos and podcast episodes. (Link 👇)

Tweet media one

27

303

2K

0

1

72

@DhruvBatraDB

Dhruv Batra

1 year

Segment Anything: general-purpose understanding of objects in images. Model+code under a liberal license following FAIR’s commitment to open research. No fear-mongering around this being unsafe for the world. Just the steady (yet fascinating) march of scientific progress.

@AIatMeta

AI at Meta

1 year

Today we're releasing the Segment Anything Model (SAM) — a step toward the first foundation model for image segmentation. SAM is capable of one-click segmentation of any object from any photo or video + zero-shot transfer to other segmentation tasks ➡️

143

2K

7K

2

12

70

@DhruvBatraDB

Dhruv Batra

8 months

Sim2real and large-scale learning (with RL) are gifts that keep on giving. And so are the reviewers at robotics conferences who aren’t yet convinced of “this whole learning thing”.

@ir413

Ilija Radosavovic

8 months

we have trained a humanoid transformer with large-scale reinforcement learning in simulation and deployed it to the real world zero-shot

95

257

2K

1

4

68

@DhruvBatraDB

Dhruv Batra

3 years

Season 2 Episode 3 is out! Kyunghyun Cho @kchonyc ( @nyuniversity , @genentech ) on Humans of AI: Stories, Not Stats. Kyunghyun talks about going with the flow while planning his day, not getting attached to past success, experiencing compassion for others, and more.

Tweet media one

2

8

68

@DhruvBatraDB

Dhruv Batra

3 years

Nervously looking forward to this talk tomorrow.

@naaclmeeting

NAACL HLT 2024

3 years

We're excited to announce #NAACL2021 's keynote speakers: Hinrich Shütze @HinrichSchuetze , Shakir Mohamed @shakir_za , Dhruv Batra @DhruvBatraDB , Thamar Solorio @thamar_solorio , Aya Soffer @asoffer , and Dan Weld @dsweld . Talk times and speaker bios here:

Tweet media one

0

19

94

1

4

67

@DhruvBatraDB

Dhruv Batra

5 years

Aishwarya Agrawal ( @gtcomputing PhD '19) was appointed as one of the CIFAR Canada AI Chairs ( @CIFAR_News ). Congratulations to @aagrawalAA and the other newly appointed chairs!

2

7

67

@DhruvBatraDB

Dhruv Batra

7 years

Excited to announce: moving forward, @deviparikh and I will split our time between Georgia Tech (GT) and Facebook AI Research (FAIR)!

1

4

66

@DhruvBatraDB

Dhruv Batra

3 years

S2E8 is out! Georgia Gkioxari @georgiagkioxari ( @facebookai ) on Humans of AI: Stories, Not Stats. Georgia talks about her love of coding, running experiments, the importance of not overthinking things, and more. At the end, I end up on the other side, answering her questions.

Tweet media one

1

9

66

@DhruvBatraDB

Dhruv Batra

3 years

Season 2 Episode 1 is out! Devi Parikh ( @deviparikh ) on Humans of AI: Stories, Not Stats. Video: Podcast: All episodes so far:

Humans of AI: Stories, Not Stats – Season 2

Getting to know AI researchers better as people. An interview series by Devi Parikh and Dhruv Batra.

www.humanstories.ai

0

7

64

@DhruvBatraDB

Dhruv Batra

6 years

Can confirm -- @abhshkdz is indeed pursuing some pretty cool research! #ProudAdvisorMoment @mlatgt @gtcomputing

@ICatGT

Georgia Tech School of Interactive Computing

6 years

"It feels within reach, the vision that we see in science fiction. Movies of robots that you can talk to or give instructions to." IC Ph.D. student @abhshkdz is pursuing some pretty cool research developing algorithms that can see, talk, and act. READ:

Tweet media one

0

5

30

1

4

63

@DhruvBatraDB

Dhruv Batra

3 years

Season 2 Episode 3 is out! Danny Tarlow @dtarlow2 ( @GoogleAI ) on Humans of AI: Stories, Not Stats. Danny talks about procrastination as a sign of burnout, making decisions based on a happiness threshold, "the score takes care of itself" philosophy, and more.

Tweet media one

1

8

62

@DhruvBatraDB

Dhruv Batra

7 years

Excited about the launch of the FAIR Residency Program! 1-year research training program designed to give talented young people from outside FB experience in cutting-edge AI research, prepare them for grad programs in ML or kickstart a research career.

Tweet media one

1

17

61

@DhruvBatraDB

Dhruv Batra

6 years

Dec 15: I have crossed the letter-of-recommendation event horizon. The requests are coming in faster than I can upload!

0

2

62

@DhruvBatraDB

Dhruv Batra

4 years

To quote @nlpnoah from his interview — I feel plenty exposed right now :-).

@deviparikh

Devi Parikh

4 years

Episode 1 is out! Dhruv Batra ( @DhruvBatraDB ) on Humans of AI: Stories, Not Stats. Video: Podcast: All episodes so far:

Tweet media one

10

39

364

0

2

61

@DhruvBatraDB

Dhruv Batra

9 months

A major milestone in robot navigation — GOAT: Go to AnyThing.

@dchaplot

Devendra Chaplot

9 months

Proud to present: GOAT: GO to AnyThing A universal navigation system that can find any object specified in any way - as an image, language, or a category - in completely unseen environments. Also useful for pick and place and social navigation! 🧵👇

5

43

302

0

7

61

@DhruvBatraDB

Dhruv Batra

6 years

Jitendra Malik speaking about how to write a good paper at the Good Citizen panel at #CVPR18 .

Tweet media one

Tweet media two

Tweet media three

Tweet media four

2

15

60

@DhruvBatraDB

Dhruv Batra

3 years

S2E12 is out! Aaron Courville ( @Mila_Quebec @UMontrealDIRO ) on Humans of AI: Stories, Not Stats. Aaron talks about his determination when chasing ideas, finding serenity in fishing, his fascination with game theory, how he treasures family time, and a lot more. [1/n]

Tweet media one

2

7

59

@DhruvBatraDB

Dhruv Batra

5 years

Someday we'll be able to write about AI without pictures of robot (killer or contemplative). But I fear we'll have "solved" AI by then.

@TechSpot

TechSpot

5 years

Facebook's Habitat generates photorealistic homes to teach AI agents real-world navigation skills

Tweet media one

0

5

2

0

2

58

@DhruvBatraDB

Dhruv Batra

3 years

Congratulations @abhshkdz ! Very well deserved and super proud!

@abhshkdz

Abhishek Das

3 years

Delighted to share that my PhD thesis won the GT Sigma Xi and College of Computing awards! Thesis pdf: GT Sigma Xi: GT CoC:

Tweet media one

Tweet media two

29

10

507

0

0

58

@DhruvBatraDB

Dhruv Batra

2 years

@tdietterich @EMostaque @paperswithcode @MetaAI “Authenticated researchers” And here I thought the promise of OSS was that anyone could access and contribute.

3

0

56

@DhruvBatraDB

Dhruv Batra

2 years

The largest dataset of real-world 3D scans with dense semantic annotations, available freely for academic use!

@AIatMeta

AI at Meta

2 years

(1/3) Today we’re releasing the Habitat-Matterport 3D Semantics dataset, the largest public dataset of real-world 3D spaces with dense semantic annotations. HM3D-Sem is free and available to use with FAIR's Habitat simulator:

5

125

511

0

8

58

@DhruvBatraDB

Dhruv Batra

4 years

This is what scaling looks like! We built @eval_ai for ourselves — to host the VQA challenge in 2017. 3 years later, we’ve hosted nearly 80 challenges from the research community, with 75k submissions from 7k teams. 14 challenges at #CVPR2020 alone. 🚀

@eval_ai

EvalAI

4 years

We are excited to share that we hosted 14 AI challenges for ongoing #CVPR2020 . Here is the list: 1. Argoverse 3D Tracking Competition @argoai 2. Argoverse Motion Forecasting Competition @argoai 3. Habitat Challenge 2020 @ai_habitat 4. RoboTHOR Challenge 2020 @allen_ai 1/4

1

6

23

1

7

58

@DhruvBatraDB

Dhruv Batra

7 years

. @deviparikh accepting her Computers and Thought award, and giving a talk at IJCAI 2017.

Tweet media one

Tweet media two

Tweet media three

0

9

57

@DhruvBatraDB

Dhruv Batra

2 years

Progress is made not by grandstanding about the right way to do AI, but by coming in every day and building things.

@eval_ai

EvalAI

2 years

New milestone: EvalAI now hosts 100+ active challenges! From 1 challenge (VQA) in 2017 to here in 5 years: - 200+ challenges - 18k+ users - 180k+ submissions - 30+ organizations

0

2

18

1

2

57

@DhruvBatraDB

Dhruv Batra

2 years

If a paper doesn't read like a "typical paper", great. Is it correct and significant? If it "reads like a blogpost", cool. Is it correct and significant? If it places the table captions below vs over the tables, why do you possibly care. Is it correct & significant?

1

0

55

@DhruvBatraDB

Dhruv Batra

3 months

Naoki Yokoyama @naokiyokoyama0 presenting his best paper award finalist talk at #ICRA2024 ! Vision-Language Frontier Maps for Zero-Shot Semantic Navigation: show how to combine VL foundation models with a mapping+search stack. @ICatGT @GTrobotics @mlatgt @BostonDynamics

Tweet media one

Tweet media two

Tweet media three

Tweet media four

@naokiyokoyama0

Naoki Yokoyama

@naokiyokoyama0

6 months

Excited to share our latest work, Vision-Language Frontier Maps – a SOTA approach for semantic navigation in robotics. VLFM enables robots to navigate and find objects in novel environments using vision-language foundation models, zero-shot! Accepted to #ICRA2024 ! 🧵

1

40

207

1

5

55

@DhruvBatraDB

Dhruv Batra

1 year

This is what a commitment to open fundamental research in AI looks like. - Llama-v2 code and models out. - APIs via Azure, AWS, HF, and others. - 7B, 13B, 70B parameters. 2T tokens. 4k context length. - Pre-trained on 40% more data than Llama-v1. - Fine-tuned on 1 million human

@ylecun

Yann LeCun

1 year

This is huge: Llama-v2 is open source, with a license that authorizes commercial use! This is going to change the landscape of the LLM market. Llama-v2 is available on Microsoft Azure and will be available on AWS, Hugging Face and other providers Pretrained and fine-tuned

423

4K

16K

0

2

54

@DhruvBatraDB

Dhruv Batra

4 years

Embodied AI workshop at #CVPR2020 - 2 day event (June 14, 15) - 12 invited talks - 3 robot nav challenges - 1 sim only - 1 sim2real (eval off-site pre-CVPR) - 1 sim2real (eval on-site at CVPR) - 33 organizers, 14 affiliations 😎

@ai_habitat

AI Habitat

4 years

Habitat is one of THREE embodied navigation challenges this year at a special 2-day workshop on Embodied AI at #CVPR2020 1. Gibson: 2. Habitat: 3. RoboThor:

Tweet media one

0

13

30

0

9

53

@DhruvBatraDB

Dhruv Batra

2 years

I’ve been fascinated by the phenomenon of emergence in philosophy and science . There’s a lot of talk in AI about world models and neuro-symbolic systems. This project gave me hope that models don’t have to be hand-designed. Models can simply emerge!

Tweet card media

Emergence - Wikipedia

en.wikipedia.org

1

3

53

@DhruvBatraDB

Dhruv Batra

1 year

Cool work by @xiaolonw ’s group. I note (with positive interest) that a robotic locomotion work is a “highlight paper” at CVPR. Speaks to a generally open-minded nature of CV venues. Similar observations have been made about NERFs appearing at CV venues not SIGGRAPH. If that

@xiaolonw

Xiaolong Wang

1 year

The robot climbs stairs🏯, steps over stones 🪨, and runs in the wild🏞️, all in one policy, without any remote control! Our #CVPR2023 Highlight paper achieves this by using RL + a 3D Neural Volumetric Memory (NVM) trained with view synthesis!

5

66

296

1

8

53

@DhruvBatraDB

Dhruv Batra

6 years

Happy to this is finally out! Great work @drewAhudson and @chrmanning ! Looking forward to state of art on this track at the VQA challenge and workshop () at #CVPR2019 . Stay tuned for EvalAI challenge page @project_cloudcv .

@stanfordnlp

Stanford NLP Group

6 years

We’ve released a new Visual Question Answering dataset to drive progress on real-image relational/compositional visual and linguistic understanding: GQA Questions, answers, images, and semantics available; will be used as a track in the VQA Challenge 2019.

Tweet media one

6

116

280

0

11

52

@DhruvBatraDB

Dhruv Batra

3 years

And that's a wrap. All S2 episodes are now out (). These were meaningful, insightful, and delightful conversations (at least for me). Huge thanks to @mkulkhanna and @VarshiniSubhash for all their help; this simply wouldn't be possible without them!

Humans of AI: Stories, Not Stats – Season 2

Getting to know AI researchers better as people. An interview series by Devi Parikh and Dhruv Batra.

www.humanstories.ai

@DhruvBatraDB

Dhruv Batra

3 years

I am excited to announce Season 2 of Humans of AI: Stories, Not Stats! @deviparikh created this series in 2020 and I am the host for Season 2, where I interview a cohort of 20 AI researchers to learn more about them.

Tweet media one

8

49

367

0

5

53

@DhruvBatraDB

Dhruv Batra

3 years

Season 2 Episode 6 is out! Judy Hoffman @judyfhoffman ( @ICatGT , @gtcomputing , @mlatgt ) on Humans of AI: Stories, Not Stats. Judy talks about her tendency to optimize every task, how she finds it rewarding to uplift those around her, the importance of family & friends, & more.

Tweet media one

1

7

53

@DhruvBatraDB

Dhruv Batra

7 years

Visual Dialog: Code, demo, code for demo. And that's where the recursion stops :-). Nice work team!

@deviparikh

Devi Parikh

7 years

Visual Dialog: code, demo (i.e., a chatbot that can see), AND code for demo: all now available at !

0

86

145

2

21

51

@DhruvBatraDB

Dhruv Batra

4 years

Hard agree with Y-Lan on this! (Wonderful interview overall)

Tweet media one

@deviparikh

Devi Parikh

4 years

Episode 5 is out! Y-Lan Boureau on Humans of AI: Stories, Not Stats. Video: Podcast: All episodes so far:

Tweet media one

0

7

79

0

2

51

@DhruvBatraDB

Dhruv Batra

6 years

. @abhshkdz presenting a talk on Embodied Question Answering at #CVPR18 , with a bold message — from static datasets to embodied agents that see, talk, act, and reason (which he’s calling a-star). With @samyakdatta , Georgia Gkioxari, Stefan Lee, @deviparikh . @mlatgt @ICatGT

Tweet media one

Tweet media two

Tweet media three

Tweet media four

1

8

51

@DhruvBatraDB

Dhruv Batra

3 years

I don't think people realize how surprising this result is: 96% success at navigating to points in new environments, no map provided OR built by the methods, no egomotion or localization sensors of any kind, noisy actuation, noisy RGBD. Pixels-in actions-out, trained at scale.

@ai_habitat

AI Habitat

3 years

Habitat Challenge 2021 results announced: . PointNav v2 saw breakthrough improvements: 15% (2019, Wijmans DD-PPO) --> 28% (2020 Challenge winners, Ramakrishnan et al) --> 96% (2021 Challenge winner, Hu et al.). [1/n]

5

2

20

1

2

49

@DhruvBatraDB

Dhruv Batra

4 months

So many robotics start-ups! Tired: thin software wrappers around ChatGPT for web agents. Wired: thin metallic wrappers around ChatGPT for robotics. Do we really need to see a humanoid robot to know that chatbots can produce engaging language?

4

1

50

@DhruvBatraDB

Dhruv Batra

5 years

On that note, @deviparikh and I are looking for research scientists and post-docs to join us at @ICatGT and @mlatgt . @panderson_me (GT -> RS @GoogleAI ), Stefan Lee (GT -> Faculty @OregonState ), and Zhile Ren (GT -> TBA) have set a high bar though 🙂.

@panderson_me

Peter Anderson

5 years

Excited to share that, starting in January, I'll be joining @GoogleAI as a Research Scientist in Austin! Looking forward to working with @jasonbaldridge , Radu Soricut, @irrfaan and others on vision and language problems, grounded language, embodied AI, etc.

12

5

190

1

9

49

@DhruvBatraDB

Dhruv Batra

1 year

Work led by FAIR @MetaAI in collaboration with a broad group of collaborators at @gtcomputing ( @mlatgt ), @Penn , @Stanford , @UCBerkeley . Visual Cortex led by: @arjunmajum @KarmeshYadav Sergio Arnaud Adaptive Skill Coordination led by: @naokiyokoyama0

Tweet media one

Tweet media two

2

5

48

@DhruvBatraDB

Dhruv Batra

3 years

S2E19 is out! Charles Isbell @isbellHFh ( @GeorgiaTech , @gtcomputing , @ICatGT ) on Humans of AI: Stories, Not Stats. Charles talks about thinking of failures as learning experiences, the difference between empathy & sympathy, the importance of long-term vision, & more. [1/n]

Tweet media one

1

10

48

@DhruvBatraDB

Dhruv Batra

4 years

My talk at #ICRA2020 workshop on Perception, Action, and Learning. How far can we scale model-free RL for Embodied AI? Turns out, surprisingly far.

@lucacarlone1

Luca Carlone

4 years

I'm excited to announce the second invited talk for PAL 2020: Dhruv Batra (Georgia Tech | #facebook AI Research | ) talks about “How far can we scale model-free RL for embodied AI”: #robotics #RL #icra2020 #pal2020ws @DhruvBatraDB

0

0

16

1

12

47

@DhruvBatraDB

Dhruv Batra

5 years

Chris Manning ( @stanfordnlp ) speaking at the VQA workshop about “Making the L in VQA Matter” (a play on @yashgoyal_ ’s making the V in VQA matter paper). And why more people (particularly non-vision people) should be working on VQA.

Tweet media one

Tweet media two

Tweet media three

Tweet media four

1

10

45

@DhruvBatraDB

Dhruv Batra

2 years

So why/how can blind AI agents navigate? Memory. Memoryless agents completely fail (0% success). LSTM-agents remember over 1000 past steps! And their memories contain collision detection neurons!

Tweet media one

Tweet media two

1

7

47

@DhruvBatraDB

Dhruv Batra

2 years

@davidchalmers42 That the best way to train robots is in simulation. And more generally, that the world of bits scales better/easier than the world of atoms. So the more we can leverage the world of bits (language models, videos, simulation), the better our efforts will be in the world of atoms

1

3

46

@DhruvBatraDB

Dhruv Batra

3 years

Season 2 Episode 7 is out! Andrew Fitzgibbon @Awfidius ( @MSFTResearch , @MSFTResearchCam ) on Humans of AI: Stories, Not Stats. Andrew talks about his love for Formula racing and skiing, his fascination with coding and prototyping, his optimistic attitude towards life and more.

Tweet media one

4

5

46

@DhruvBatraDB

Dhruv Batra

7 years

Learning Cooperative Visual Dialog Agents with Deep RL @abhshkdz * @SatwikKottur * José Moura Stefan Lee Dhruv Batra

1

19

45

@DhruvBatraDB

Dhruv Batra

7 years

Jiasen Lu's spotlight at #CVPR17 on image captioning; reasoning about which words should be grounded -- where to look AND when to look!

Tweet media one

Tweet media two

1

8

45

@DhruvBatraDB

Dhruv Batra

3 years

3 years ago, a group of us got together to study benchmarking in Embodied AI and robotics. The result was the SPL metric by @panderson_me et al. Here is SPL revisited for real robots and informed by what we have learned from sim2real transfer.

@naokiyokoyama0

Naoki Yokoyama

@naokiyokoyama0

3 years

How can we measure the navigation performance of robots with various dynamics? One way is by path length. But the shortest path is not always the fastest if the robot can move and turn at the same time, which most real robots can (LoCoBot, Fetch, etc).

3

8

42

1

5

45

@DhruvBatraDB

Dhruv Batra

8 months

@AjdDavison I’m happy to take the other side of that bet if you want to make it precise. If data isn’t the bottleneck and all we needed was human ingenuity, well, we had 40+ years to do that. And all we got were cute stories with mediocre results.

1

1

42

@DhruvBatraDB

Dhruv Batra

5 years

I’ve always viewed concerns about linguistic biases in V+L datasets as temporary hurdles at best and unproductive grandstanding at worst. You can recite witty stories about clever-hans all you want, but you can’t argue with progress; (quantitative: plot, qualitative: demo).

@deviparikh

Devi Parikh

5 years

Tweet media one

0

0

21

4

9

43

@DhruvBatraDB

Dhruv Batra

5 years

Measuring sim2real generalization: 3D scan a real env, run parallel studies in sim and reality, measure correlation. Of course RL agents learn to cheat in simulation! But it can be overcome.

@joannetruong

Joanne Truong

5 years

We 3D scan a lab and create a virtualized replica in simulation. This allows us to run parallel experiments in simulation and reality — at scale (810 identical experiments)!

1

5

24

1

5

44

@DhruvBatraDB

Dhruv Batra

2 years

We trained blind AI agents to navigate. No vision, audio, olfactory, magnetic, or any other sensing (as in animals). Just egomotion - how much did I just move? GPS+Compass in EAI. Can blind AI agents navigate? Yes! 95% success. How? By learning to follow walls and obstacles.

Tweet media one

1

3

43

@DhruvBatraDB

Dhruv Batra

9 months

I’ve always wondered why RL is a sub-community with a distinct identify from machine learning. We don’t have a conference on SSL or on supervised learning, why RL? Imagine how bizarre “The International Conference on Kmeans” would sound? Someone help me see their

@RL_Conference

RL_Conference

9 months

Thrilled to announce the first annual Reinforcement Learning Conference @RL_Conference , which will be held at UMass Amherst August 9-12! RLC is the first strongly peer-reviewed RL venue with proceedings, and our call for papers is now available: .

Tweet media one

4

95

239

11

2

43

@DhruvBatraDB

Dhruv Batra

5 years

An example of how quickly AI research is progressing. — Feb ’18: Workshop on Embodied AI at FAIR. We struggle to define what Embodied AI even is. — Jul ’18: Workshop working group defines PointGoalNav and SPL (). — Feb ’19: @ai_habitat released. 1/n

Tweet card media

On Evaluation of Embodied Navigation Agents

Skillful mobile operation in three-dimensional environments is a primary topic of study in Artificial Intelligence. The past two years have seen a surge of creative work on navigation. This...

@AIatMeta

AI at Meta

5 years

Facebook AI has effectively solved the task of point-goal navigation by AI agents in simulated environments, using only a camera, GPS, and compass data. Agents achieve 99.9% success in a variety of virtual settings, such as houses and offices.

1

104

339

1

8

43

@DhruvBatraDB

Dhruv Batra

2 years

My students and collaborators ( @ICatGT @gtcomputing @MetaAI ) are presenting advances in embodied AI at #NeurIPS2022 this week. A summary 🧵

1

6

43