Jonathon Luiten @JonathonLuiten Twitter profile

Pinned Tweet

Jonathon Luiten

1 year

Dynamic 3D Gaussians: Tracking by Persistent Dynamic View Synthesis We model the world as a set of 3D Gaussians that move & rotate over time. This extends Gaussian Splatting to dynamic scenes, with accurate novel-view synthesis and dense 3D trajectories.

26

386

2K

Last Seen Profiles

@eAzW17QhhK15t

@ScienceClic

@suhyeon33504

@diegonunee

@Too6Lolo

@chillindreamies

@Thomas_J_West

@AndrsColman3

@JikeSpingleton

@massilia135_

@BRP_JJCC

@salam_464

@iheartgyyu

@SunderlandUy

@TeixeiraxxD

@RezeIsBombDevil

@luke_ella35328

@bokeplokalmalam

@Gahlotceo

@tjin1647882

@DealNews

@RickyHeath88402

@bokeplokalmalam

@Jeffloyh1

@moco_nya

@BlindBatBrewer

@MythicMayansNFT

@rknly01130937

@DDMissTaylor

@Draaxus24

@SipherDan

@Sophia406464631

@PhoenixNightsNC

@Martin_CSP

@friendsofrss

@jisseunz

Jonathon Luiten

@JonathonLuiten

1 year

Code is released!!!!! Been getting ALOT of requests for this (450+ github stars before code is even released???)

GitHub - JonathonLuiten/Dynamic3DGaussians

Contribute to JonathonLuiten/Dynamic3DGaussians development by creating an account on GitHub.

github.com

Jonathon Luiten

@JonathonLuiten

1 year

Dynamic 3D Gaussians: Tracking by Persistent Dynamic View Synthesis We model the world as a set of 3D Gaussians that move & rotate over time. This extends Gaussian Splatting to dynamic scenes, with accurate novel-view synthesis and dense 3D trajectories.

26

386

2K

16

131

846

Jonathon Luiten

@JonathonLuiten

3 years

For all you NeRF people: Instant-NGP (the crazy fast train a NeRF in 5 second method), has released pytorch bindings (which make it MUCH easier to use, compared to raw CUDA code).

4

35

389

Jonathon Luiten

@JonathonLuiten

2 years

Excited to present our @CVPR Oral paper HODOR Typically, Video Object Segmentation methods learn low-level pixel correspondence. Instead, we use transformers to extract high-level object embeddings, that can be used to re-segment objects through video.

1

52

307

Jonathon Luiten

@JonathonLuiten

10 months

This is my first follow-up work after Dynamic 3D Gaussians. Nikhil ( @Nik__V__ ) and I have been working closely to build a system that can tackle SLAM using Gaussian Splatting – estimating camera poses without COLMAP and working in real-time on live streaming data.

Nikhil Keetha

@Nik__V__

10 months

SplaTAM: Splat, Track & Map 3D Gaussians for Dense RGB-D SLAM We extend Gaussian Splatting to solve SLAM, i.e., automatically calculate the camera poses when fitting the Gaussian scene from RGB-D videos. Try it on your own iPhone capture today! 🧵👇

26

105

571

5

24

208

Jonathon Luiten

@JonathonLuiten

1 year

This enables a number of exciting applications such as the composition of different dynamic scene elements, first-person view synthesis and temporally consistent 4D scene editing. It’s also FAST! These render at 850 FPS, and only take around 2 hours to train on a single GPU.

6

26

198

Jonathon Luiten

@JonathonLuiten

1 year

I also made an interactive dynamic 3D viewer. I honestly think this is going to be the future of all of entertainment. Movies + Games converging to the same the thing. The future will for sure be 'Dynamic' and '3D', and my bet is on it being made of Gaussians.

Jonathon Luiten

@JonathonLuiten

1 year

Dynamic 3D Gaussians: Tracking by Persistent Dynamic View Synthesis We model the world as a set of 3D Gaussians that move & rotate over time. This extends Gaussian Splatting to dynamic scenes, with accurate novel-view synthesis and dense 3D trajectories.

26

386

2K

5

25

147

Jonathon Luiten

@JonathonLuiten

1 year

Interesting fact: I was working on Dynamic 3D Gaussians for a while before the 3D Gaussian Splatting paper came out. Originally I used the “fuzzy metaball” Gaussians from @leo_nid_k but swapped to splat version for fast cuda code. Def check out Leo’s paper

Approximate Differentiable Rendering with Algebraic Surfaces

Differentiable renderers provide a direct mathematical link between an object's 3D representation and images of that object. In this work, we develop an approximate differentiable renderer for a...

arxiv.org

Jonathon Luiten

@JonathonLuiten

1 year

Dynamic 3D Gaussians: Tracking by Persistent Dynamic View Synthesis We model the world as a set of 3D Gaussians that move & rotate over time. This extends Gaussian Splatting to dynamic scenes, with accurate novel-view synthesis and dense 3D trajectories.

26

386

2K

4

7

132

Jonathon Luiten

@JonathonLuiten

4 years

Multi-Object Tracking (MOT) has been notoriously difficult to evaluate, and evaluation has been a constant source of frustration for many. Check out this blog post ( ) which describes our recent work on the HOTA metrics for better tracking evaluation! 1/6

How to evaluate tracking with the HOTA metrics

HOTA (Higher Order Tracking Accuracy) is a novel metric for evaluating multi-object tracking (MOT) performance. It is designed to overcome…

jonathonluiten.medium.com

1

21

81

Jonathon Luiten

@JonathonLuiten

1 year

Giving a talk on this work at ICCV at 9am on Tuesday at the NeRF4ADR workshop (). See you all there!

NeRF4ADR: Neural Fields for Autonomous Driving and Robotics

ICCV2023 Worshop NeRF4ADR: Neural Fields for Autonomous Driving and Robotics.

neural-fields.xyz

Jonathon Luiten

@JonathonLuiten

1 year

Dynamic 3D Gaussians: Tracking by Persistent Dynamic View Synthesis We model the world as a set of 3D Gaussians that move & rotate over time. This extends Gaussian Splatting to dynamic scenes, with accurate novel-view synthesis and dense 3D trajectories.

26

386

2K

2

5

72

Jonathon Luiten

@JonathonLuiten

2 years

Can we track objects for which we don’t have training data? Check out our @CVPR 22 Oral paper Opening Up Open World Tracking We present a benchmark, baseline & analysis for kickstarting open-world tracking! Oral / Poster Friday #CVPR #CVPR22 #CVPR2022

1

12

69

Jonathon Luiten

@JonathonLuiten

1 year

Check out the website and paper! There is A LOT more cool videos and results to explore! Website: Paper: Also thanks to my collaborators George Kopanas ( @GKopanas ), Bastian Leibe ( @RWTHVisionLab ) and Deva Ramanan ( @RamananDeva ).

1

9

69

Jonathon Luiten

@JonathonLuiten

3 years

RobMOTS: The Ultimate Tracking and Video Segmentation Challenge at CVPR'21. Deadline June 11th. 8 different benchmarks come together, to create the ultimate combined challenge. Waymo, KITTI, BDD100K, TAO, MOTChallenge, YouTube-VOS, OVIS and DAVIS.

2

21

61

Jonathon Luiten

@JonathonLuiten

1 year

3D tracking results are on average only 1.5cm off from the ground-truth, in fast-moving complex scenes 150 frames long. All while densely tracking around 300k Gaussians simultaneously. See the comparison between our tracks for certain points (blue) and the ground-truth (red).

4

7

60

Jonathon Luiten

@JonathonLuiten

1 year

Wow! Rendering 3D Gaussian scenes at 60FPS on my iPhone 11. This will work for my dynamic version too! Very exciting!

Kevin Kwok

@antimatter15

1 year

I implemented 3D Gaussian Splatting in WebGL. Check it out here:

34

172

1K

4

1

56

Jonathon Luiten

@JonathonLuiten

2 years

Forecasting object locations directly from raw LiDAR is hard. In our @CVPR paper, FutureDet, we repurpose 3D detection architectures for forecasting, by directly predicting ‘Future Object Detections’ #CVPR #CVPR2022 #CVPR22

1

13

56

Jonathon Luiten

@JonathonLuiten

2 years

Excited to present 3 papers (w 2 Orals) this week @CVPR 1 - HODOR: Video Segmentation trained without Video 2 - Open World Tracking: Tracking objects classes beyond those in training 3 - FutureDet: Reformulating forecasting as Future Detection 🧵👇 #CVPR #CVPR2022 #CVPR22

3

7

53

Jonathon Luiten

@JonathonLuiten

1 year

The core idea is enforcing that Gaussians have persistent color, opacity, and size over time; and regularizing Gaussians' motion and rotation with local-rigidity constraints. Dense 6-DOF tracking emerges from persistent dynamic view synthesis, without correspondence or flow input

1

51

Jonathon Luiten

@JonathonLuiten

1 year

@svpino @mlhobbyist Matlab?

2

0

47

Jonathon Luiten

@JonathonLuiten

3 years

4 days until the Robust Video Scene Understanding Workshop at CVPR! 8 exciting invited speakers including @FidlerSanja @kkitani @judyfhoffman @WeidiXie , Katerina Fragkiadaki, Philipp Krähenbühl, Michael Felsberg and Lorenzo Torresani. + MUCH MORE.. 🧵👇

2

22

45

Jonathon Luiten

@JonathonLuiten

4 years

Interested in Object Tracking in 3D? Dynamic Object Reconstruction? Generic Object Discovery? Open Set Scene Understanding? I am currently presenting two papers on these topics at (virtual) ICRA 2020. @ICRA2020 #ICRA2020 #ICRA Join the discussion! 1/5

2

14

39

Jonathon Luiten

@JonathonLuiten

3 years

Details of this can be found here: I don't have any more time to dedicate to this, but if someone can get this to work and let me know that would be awesome!

How to use underlying representation for a new task with different input/output dimensions? · Issue...

This code and project are awesome! Thanks a lot. In terms of building upon this, I wonder how to access, edit and train the underlying hash+NN representation for a new task. For example, let's ...

github.com

1

2

31

Jonathon Luiten

@JonathonLuiten

3 years

Opening up Open-World Tracking. It’s impossible to label EVERY class an agent might see. But not detecting and tracking UNKNOWN objects, may lead to DISASTER. In Open-World Tracking, trackers trained only 80 classes need to track ANY unknown object.

1

6

31

Jonathon Luiten

@JonathonLuiten

1 year

Well done Luma team! I’m impressed by how fast you shipped this. Now time to make it dynamic?

Luma AI

@LumaLabsAI

1 year

3D was either pretty, or fast. Now it’s BOTH! Meet Interactive Scenes built with Gaussian Splatting: 🔥Browser & Phone-Friendly: Hyperefficient and fast rendering everywhere 👌Embed Anywhere: 8-20MB streaming files (even smaller soon!) ✨Ultra High Quality offline NeRF renders &

57

335

2K

1

0

23

Jonathon Luiten

@JonathonLuiten

4 years

Huge Update!!! HOTA Metrics now evaluated live on the MOTChallenge benchmark too! And TrackEval code now the official evaluation code for MOTChallenge. Check it out: Another huge step toward the future of tracking research.

1

6

19

Jonathon Luiten

@JonathonLuiten

4 years

I'm giving a talk about Visual Object Tracking in about 2.5 hours. I will cover advances from old-school Lucas-Kanade Template Tracking to our state-of-the-art Siam R-CNN (). Sign up to attend here:

2

21

Jonathon Luiten

@JonathonLuiten

1 year

@rerundotio I have a question for you guys! Do you think I could replace open3D in my current vis pipeline with rerun? Can rerun render a 300k point point cloud which is updated every timestep, at the 800fps I can create them? Details and implementation here: See

GitHub - JonathonLuiten/Dynamic3DGaussians

Contribute to JonathonLuiten/Dynamic3DGaussians development by creating an account on GitHub.

github.com

3

0

17

Jonathon Luiten

@JonathonLuiten

1 year

@dylan_ebert_ This is a fantastic video. Would love to see one for my dynamic Gaussians.

1

0

17

Jonathon Luiten

@JonathonLuiten

1 year

My suggestion to fix reviewing: instead of individuals reviewing have teams of authors from other submissions review. The joint authors of each submission need to review 3 papers together as a team. Potentially if they neglect their reviewing duties their submission will be desk

David Picard

@david_picard

1 year

3 times as many authors as there are reviewers. This is obviously a problem, but how do we solve it? #ICCV

13

2

44

3

0

17

Jonathon Luiten

@JonathonLuiten

1 year

@s1ddok Wholey moley this is cool!!!! We should def chat! I want to see the dynamic stuff running on a VR headset so bad! Also putting the view dependent effects back in shouldn’t make it any slower. It was 850 fps because I was only doing 640x360 ims.

1

0

16

Jonathon Luiten

@JonathonLuiten

4 years

The workshop I am organizing on Multi-Object Tracking and Segmentation has just begun! Tune in live: (or join in on zoom - link on CVPR workshop landing page) Exciting talks from Bernt Schiele, Raquel Urtasun, Xin Wang and Alyosha Efros

0

2

15

Jonathon Luiten

@JonathonLuiten

4 years

Calling all object tracking researchers!!! Submit your trackers to our ECCV workshop challenge on Tracking Any Object (TAO) (tracking on 833 categories!) Deadline is August 16!!! Excited to see you all present your results at our Workshop!

0

4

15

Jonathon Luiten

@JonathonLuiten

1 year

@VGolyanik SceneNeRFlow is very cool work. Lots of things really similar. Literally 9 min before I uploaded my gf sent me it saying “scoopy doopy do”. I would love to compare it to Dynamic 3D Gaussians, on both the dataset I use and the dataset they use!! @tretschk @MZollhoefer @chlassner

0

14

Jonathon Luiten

@JonathonLuiten

1 year

@Shedletsky @CoffeeVectors The 'input' is NOT the video you see. It's actually a bunch of static cameras. Here we are reconstructing the dynamic world in a persistent way across time with a bunch of small Gaussians. This enables us to render novel views (e.g. the loop you see) and also to track all...

2

1

10

Jonathon Luiten

@JonathonLuiten

2 years

This enables some awesome behavior, such as being able to train WITHOUT VIDEO (from single images), or from video where only one frame is labeled. Oral talk and poster Tuesday afternoon (NOW) – in the Video Analysis session. #CVPR #CVPR22 #CVPR2022

2

1

13

Jonathon Luiten

@JonathonLuiten

1 year

I just learnt I'm a Shape Rotator

Jonathon Luiten

@JonathonLuiten

1 year

Dynamic 3D Gaussians: Tracking by Persistent Dynamic View Synthesis We model the world as a set of 3D Gaussians that move & rotate over time. This extends Gaussian Splatting to dynamic scenes, with accurate novel-view synthesis and dense 3D trajectories.

26

386

2K

1

0

13

Jonathon Luiten

@JonathonLuiten

3 years

Workshop begins in ~1.5 hours at 7am EDT / 1pm CEST. Don't miss all the invited speakers + papers + challenge results! Featured in #CVPR Daily magazine! YouTube livestream open to everyone: Schedule here:

CVPR Robust Video Scene Understanding Workshop

Live stream of the Robust Video Scene Understanding Workshop @ CVPR.See webpage for schedule + details:https://eval.vision.rwth-aachen.de/rvsu-workshop21/

www.youtube.com

Jonathon Luiten

@JonathonLuiten

3 years

4 days until the Robust Video Scene Understanding Workshop at CVPR! 8 exciting invited speakers including @FidlerSanja @kkitani @judyfhoffman @WeidiXie , Katerina Fragkiadaki, Philipp Krähenbühl, Michael Felsberg and Lorenzo Torresani. + MUCH MORE.. 🧵👇

2

22

45

1

9

13

Jonathon Luiten

@JonathonLuiten

2 years

@keenanisalive Infinite Nature: Perpetual View Generation of Natural Scenes from a Single Image @akanazawa @Jimantha

Infinite Nature: Perpetual View Generation of Natural Scenes from a Single Image

Learning to generate arbitrarily long videos of aerial trajectories through nature scenes.

infinite-nature.github.io

2

1

13

Jonathon Luiten

@JonathonLuiten

1 year

@CoffeeVectors It only uses so much VRAM because it stores the training images on the GPU by default. But there is a toggle to turn this off and store them on CPU until needed. With this the amount of VRAM is VERY small.

1

0

13

Jonathon Luiten

@JonathonLuiten

1 year

@CorahMicah I’m EXTREMELY interested. There is a lot of follow up work to do, but it’ll take the whole community. I’m excited to release the code so people can play with it. Also happy to collaborate and help people out with their own projects building on this.

3

0

11

Jonathon Luiten

@JonathonLuiten

1 year

@eigenhector I think it’s really not that hard. You can see my PR on the Gaussian splatting paper which adapts the cuda code to also render depth. I have another internal version which renders median depth instead of mean depth which I find gives better geometry (no bleeding between edges)

1

2

11

Jonathon Luiten

@JonathonLuiten

3 years

Currently the bindings are for the underlying hash representation and super fast mlps, and there is a pytorch example for fitting to a 2D image. I spent today trying to incorporate this into NeRF, with a little, but mostly not very much, success.

1

0

11

Jonathon Luiten

@JonathonLuiten

4 years

Trying to improve evaluation of Multi-Object Tracking and need your help in trying to judge what makes a good tracker. Anyone can help, and any of your time would be appreciated.

1

7

10

Jonathon Luiten

@JonathonLuiten

1 year

@AjdDavison @alzugarayign Thanks Andrew! I remember meeting you in Sicily in I think 2018. I was and still am in awe of all your amazing work. Means a lot to me that you also like my work!

1

0

9

Jonathon Luiten

@JonathonLuiten

4 years

We have released a tracking evaluation codebase: TrackEval. It contains HOTA + many other metrics, and runs on multiple benchmark formats. It is 100% python, easy to understand and extend, and SUPER FAST (10x faster than previous evaluation code). 5/6

GitHub - JonathonLuiten/TrackEval: HOTA (and other) evaluation metrics for Multi-Object Tracking...

HOTA (and other) evaluation metrics for Multi-Object Tracking (MOT). - JonathonLuiten/TrackEval

github.com

1

8

Jonathon Luiten

@JonathonLuiten

3 years

Also: RVSU CVPR'21 Workshop Call for Papers. Call for submission track papers on Tracking, Video Segmentation and other aspects of Video Understanding. Deadline June 4th. Paper restricted to 4 pages to allow joint submission at main track conferences.

1

4

9

Jonathon Luiten

@JonathonLuiten

3 years

8.5 days to submit your trackers to the ultimate tracking challenge. RobMOTS evaluating multi object tracking across 8 different benchmarks. Val and test servers now live.

1

4

8

Jonathon Luiten

@JonathonLuiten

10 months

@LvZhaoyang GPS-Gaussian

0

8

Jonathon Luiten

@JonathonLuiten

1 year

@Eulerson314 It’s all this guy, and it always will be:

2

1

8

Jonathon Luiten

@JonathonLuiten

4 years

Previous metrics either overemphasize detection (MOTA), or association (IDF1) while mostly ignoring the other. HOTA is designed to evenly balance between both of these. More details in our IJCV paper (open access): 3/6

1

7

Jonathon Luiten

@JonathonLuiten

1 year

This awesome work from my colleague Andrew was the motivation behind my “Gaussian-eye view” renderings. Definitely check it out!

Chonghyuk (Andrew) Song

@ndsong95

1 year

Ever wondered what the world looks like to your pet dog? Our latest #ICCV2023 paper, Total-Recon, enables embodied view synthesis of deformable scenes from a casual RGBD video: Drop by poster #10 on Friday 10:30~12:30pm in Rm. Foyer Sud to know more! 1/2

1

22

110

0

1

7

Jonathon Luiten

@JonathonLuiten

10 months

Check out this super cool work from Joanna @materzynska

Joanna

@materzynska

10 months

In our new paper, "Customizing Motion in Text-to-Video Diffusion Models" we show a method for introducing novel motions into text-to-video diffusion models. Given a few examples of a novel motion and a generic description, our method creates a new text mapping in the network.

2

11

79

0

7

Jonathon Luiten

@JonathonLuiten

4 years

Joint work together with @AljosaOsep , @PatrickDendorf1 , @OxfordTVG , Andreas Geiger, @lealtaixe , and @RWTHVisionLab . And don’t forget to check out the blog post: 6/6

How to evaluate tracking with the HOTA metrics

HOTA (Higher Order Tracking Accuracy) is a novel metric for evaluating multi-object tracking (MOT) performance. It is designed to overcome…

jonathonluiten.medium.com

2

1

6

Jonathon Luiten

@JonathonLuiten

3 years

Organized jointly together with: @MOTChallenge @Waymo @RWTHVisionLab @lealtaixe @DrFisherYu @TobiasFischer11 @AljosaOsep @jvlmdr @AlexBewleyAI @achalddave @tarashakhurana @XinggangWang @yuning_chai @jponttuset @PatrickDendorf1 @skprat @Istvan_Sarandi + more not on twitter.

1

2

7

Jonathon Luiten

@JonathonLuiten

1 year

If someone can get this on the front page of hacker news I'll buy them a beer!

Jonathon Luiten

@JonathonLuiten

1 year

Dynamic 3D Gaussians: Tracking by Persistent Dynamic View Synthesis We model the world as a set of 3D Gaussians that move & rotate over time. This extends Gaussian Splatting to dynamic scenes, with accurate novel-view synthesis and dense 3D trajectories.

26

386

2K

1

6

Jonathon Luiten

@JonathonLuiten

1 year

@savvyRL I think this is more human bias. I think human AC's are doing most of the heavy lifting here vs a matching algo. Ofc the AC's suggestions are influenced by the matching algos recommendations also.

1

0

1

Jonathon Luiten

@JonathonLuiten

4 years

HOTA is calculated by combining three IoU scores: one for each of detection, association and localization! HOTA combines these into one score, while also allowing analysis of each, and further dividing each of these into a recall and precision component. 2/6

1

0

5

Jonathon Luiten

@JonathonLuiten

1 year

Very cool new work from my colleague Chonghyuk (Andrew) Song!!!

AK

@_akhaliq

1 year

Total-Recon: Deformable Scene Reconstruction for Embodied View Synthesis abs: project page:

1

22

146

0

6

Jonathon Luiten

@JonathonLuiten

10 months

@fdellaert All (at least) 6 papers (so far)

1

0

6

Jonathon Luiten

@JonathonLuiten

1 year

@savvyRL I should read tweets more carefully hahahhaa.

0

1

Jonathon Luiten

@JonathonLuiten

1 year

@jon_barron I prefer to just call them “3D Gaussians”. Differentiates a bit from MLP based representations. Agree a “splat” is not good.

1

0

4

Jonathon Luiten

@JonathonLuiten

1 year

@MrCatid @leo_nid_k Thanks a lot! Yeah I’m super excited about the applications of this! Lots of cool stuff is possible!

0

4

Jonathon Luiten

@JonathonLuiten

3 years

Panel session starting in 5 mins. Including: @FidlerSanja , @kkitani , @judyfhoffman , @WeidiXie , and Philipp Krähenbühl. (open for everyone). Or through #CVPR website for the zoom link to ask questions live.

CVPR Robust Video Scene Understanding Workshop

Live stream of the Robust Video Scene Understanding Workshop @ CVPR.See webpage for schedule + details:https://eval.vision.rwth-aachen.de/rvsu-workshop21/

www.youtube.com

Jonathon Luiten

@JonathonLuiten

3 years

4 days until the Robust Video Scene Understanding Workshop at CVPR! 8 exciting invited speakers including @FidlerSanja @kkitani @judyfhoffman @WeidiXie , Katerina Fragkiadaki, Philipp Krähenbühl, Michael Felsberg and Lorenzo Torresani. + MUCH MORE.. 🧵👇

2

22

45

0

5

Jonathon Luiten

@JonathonLuiten

1 year

@GKopanas It was great learning from you too @GKopanas

0

5

Jonathon Luiten

@JonathonLuiten

1 year

@Shedletsky @CoffeeVectors yes there are 27 input cameras in a semi-circular dome facing inward, and we accurately know the positions of them. This definitely makes the problem easier compared to having fewer cameras and not knowing where they are. There is lots of interesting future work to do for sure!

2

0

4

Jonathon Luiten

@JonathonLuiten

3 years

If people want to contribute to this, we're building this code open source together here:

GitHub - JonathonLuiten/instant-nerf-pytorch: NeRF trained SUPER FAST in pytorch

NeRF trained SUPER FAST in pytorch. Contribute to JonathonLuiten/instant-nerf-pytorch development by creating an account on GitHub.

github.com

1

0

5

Jonathon Luiten

@JonathonLuiten

4 years

Is there is an app that exports RGB-D video from IPad Pro w/LiDAR. Don't want meshes or pt clouds. Just RGB video + greyscale depth video. Lots of scanning apps. None give RGB-D. Could write exporter from but better if don't have to. Maybe @nobbis knows?

Creating a Fog Effect Using Scene Depth | Apple Developer Documentation

Apply virtual fog to the physical environment.

developer.apple.com

1

5

Jonathon Luiten

@JonathonLuiten

1 year

@eigenhector Eg I understand why one would want to mesh an mlp based nerf - because mlps suck to deal with. But Gaussians already intrinsically have all the properties one would want from a geometry representation in my opinion.

0

5

Jonathon Luiten

@JonathonLuiten

3 years

Huge thanks to @Google ( @GoogleAI ), @Facebook ( @facebookai ) and @CVL_ETH for agreeing to sponsor the workshop and challenge.

0

5

Jonathon Luiten

@JonathonLuiten

4 years

I have loved living, studying and researching in Germany for the last 3.5 years. For some of the reasons why this has been so great see this thread.

Andreas Zeller

@AndreasZeller

4 years

What's it like to be a #PhD student in #Germany ? You can get paid, and well. You can afford a car, an apartment, and provide for a family. You may work with great advisors at great institutions. And the food... well, the food. Read on!

59

472

2K

0

5

Jonathon Luiten

@JonathonLuiten

2 years

@keenanisalive @akanazawa @Jimantha This requires a single image as input to generate and infinite perceptual scene. But that single image could be generated from a standard image GAN so then it could be fully automatic.

0

5

Jonathon Luiten

@JonathonLuiten

4 years

HOTA has now launched live as the official metrics for KITTI tracking and KITTI MOTS. This will open many new opportunities for developing trackers. Tracking: MOTS: 4/6

1

0

3

Jonathon Luiten

@JonathonLuiten

4 years

Paper: (ArXiv: ) Video: (YouTube: ) Slack: (Channel: #wed16_5 ) With Tobias Fischer and Bastian Leibe. 3/5

Track to Reconstruct and Reconstruct to Track (MOTSFusion) - ICRA2020...

Video describing our 2020 paper on using 3D reconstruction for improving object tracking. Published in Robotics and Automation Letters 2020 (RA-L), and prese...

www.youtube.com

1

0

4

Jonathon Luiten

@JonathonLuiten

1 year

@s1ddok hahaha thanks! I also keep thinking of things to add! Now a note about seg masks I think... stay tuned...

0

4

Jonathon Luiten

@JonathonLuiten

1 year

@miguel_algaba @JSelikoff Yeah! At the moment I have only run it on scenes with multiples cameras (these real scenes have 27 train cameras, and some synthetic scenes have 20). I don’t know how well it would work with less. Also note calibrating multiple cameras in the wild is really hard.

1

0

5

Jonathon Luiten

@JonathonLuiten

1 year

@giffmana Hahaha these Gaussians aren’t too different to triangles tbh. The magic is that they are nice and differentiable so that we can easily fit them from real data with diff rendering.

1

0

4

Jonathon Luiten

@JonathonLuiten

1 year

@smallfly @AceOfThumbs @Scobleizer I am open sourcing the splatting viewer I built in the next few days. Should be very easy to input a list of extrinsic matrices and output a video. Or alternatively control the camera path interactively. Surprising to hear that INGP training is faster for you. This can be fixed.

1

0

4

Jonathon Luiten

@JonathonLuiten

3 years

Joint work with @AljosaOsep , @achalddave , @lealtaixe , @RWTHVisionLab (Bastian Leibe), Deva Ramanan, Idil Esen Zulfikar and Yang Liu.

0

4

Jonathon Luiten

@JonathonLuiten

1 year

@danfei_xu Thanks for you nice words! It also blows me away how well it works!

0

4

Jonathon Luiten

@JonathonLuiten

3 years

Thanks to @kangle_deng for contributing to this code today and fixing some of my bugs! Thanks to @yen_chen_lin for the great NeRF pytorch implementation which makes it easy to build upon. Finally, thanks to Thomas Müller for the super fast CUDA code and pytorch bindings.

0

4

Jonathon Luiten

@JonathonLuiten

3 years

Kris Kitani ( @kkitani ) is live now. Giving a keynote on "Perception + Prediction for autonomous driving." (open for everyone). Or through #CVPR website for the zoom link.

CVPR Robust Video Scene Understanding Workshop

Live stream of the Robust Video Scene Understanding Workshop @ CVPR.See webpage for schedule + details:https://eval.vision.rwth-aachen.de/rvsu-workshop21/

www.youtube.com

Jonathon Luiten

@JonathonLuiten

3 years

4 days until the Robust Video Scene Understanding Workshop at CVPR! 8 exciting invited speakers including @FidlerSanja @kkitani @judyfhoffman @WeidiXie , Katerina Fragkiadaki, Philipp Krähenbühl, Michael Felsberg and Lorenzo Torresani. + MUCH MORE.. 🧵👇

2

22

45

0

1

4

Jonathon Luiten

@JonathonLuiten

1 year

@hegemonetics hahaha yeah 850 fps is ALOT. But note that these are 640x360 images. It's a bit slower at full HD but it's still REALLY fast! 1280x720: 400 fps 1920x1080: 250 fps

0

4

Jonathon Luiten

@JonathonLuiten

1 year

@kwiledirects I’m excited for when anyone can build this from a phone capture in seconds. I think we are very close to that being a reality

1

4

Jonathon Luiten

@JonathonLuiten

1 year

@dimadamen My votes for “panoptic”

0

4

Jonathon Luiten

@JonathonLuiten

4 years

Thanks for the talk invite @talks_cv . And shout out to my co-authors Paul Voigtlaender, Philip Torr ( @OxfordTVG ) and Bastian Leibe ( @RWTHVisionLab ).

1

4

Jonathon Luiten

@JonathonLuiten

4 years

@giffmana I found it! Seems to work super nice for RGB-D. Can export as mp4 and even do streaming over usb or wifi! Almost perfect. Gives intrinsics but doesn't seem to give extrinsics / camera pose... Keeping an eye out for something that does both.

2

1

4

Jonathon Luiten

@JonathonLuiten

1 year

@janusch_patas Yeah I got a bit delayed. Will try finish it Monday 9th (after ICCV) but no promises.

0

4

Jonathon Luiten

@JonathonLuiten

1 year

@janusch_patas @jonstephens85 @RobMakesMeta @Scobleizer If I prioritise code release over other things I wanted to do I might be able to release in ~1 week. Otherwise I’ll be camping in the Andes and it’ll have to wait for Oct 1 ish. But would like to get people’s hands on it.

1

0

4

Jonathon Luiten

@JonathonLuiten

1 year

@smallfly @AceOfThumbs @Scobleizer Is it possible to make a side-by-side with the same camera path (the nerf camera path was better). This would be quite illuminating. Also make sure to mention the training time and video rendering time for each. Should be significantly different.

2

0

3

Jonathon Luiten

@JonathonLuiten

1 year

@xiuming_zhang For now only this cmu panoptic dataset and a synthetic dataset from particle nerf. I would love to run it on more data though!

0

3

Jonathon Luiten

@JonathonLuiten

1 year

@janusch_patas @nobbis @JulienBlanchon @antimatter15 I think the point is the MIT licence applies to the code that they wrote (eg what is in the repo). The code that they use from other repos (eg diff Gaussian rasterisation) is not mit. Thus you can’t actually run the code commercially, but you may use the new part of the code. I

0

3

Jonathon Luiten

@JonathonLuiten

1 year

@danielpikl This is super awesome!

0

3

Jonathon Luiten

@JonathonLuiten

2 years

@CVPR I will be presenting the oral presentation at around 1.50 in the datasets track (in 40 minutes), and after that come talk to us all at the poster at 3pm (poster 35b)

0

3

Jonathon Luiten

@JonathonLuiten

2 years

We also show current end-2-end forecasting evaluation metrics and severely gameable, and present a better suite of evaluation metrics. Come check out our poster @cvpr on Friday morning!

1

0

3

Jonathon Luiten

@JonathonLuiten

2 years

Jonathon Luiten

@JonathonLuiten

2 years

Forecasting object locations directly from raw LiDAR is hard. In our @CVPR paper, FutureDet, we repurpose 3D detection architectures for forecasting, by directly predicting ‘Future Object Detections’ #CVPR #CVPR2022 #CVPR22

1

13

56

0

1

3

Jonathon Luiten

@JonathonLuiten

1 year

@Shedletsky @CoffeeVectors hahaha fair! Gaussians are the new hype!

0

2

Jonathon Luiten

@JonathonLuiten

2 years

@CVPR

Jonathon Luiten

@JonathonLuiten

2 years

Excited to present our @CVPR Oral paper HODOR Typically, Video Object Segmentation methods learn low-level pixel correspondence. Instead, we use transformers to extract high-level object embeddings, that can be used to re-segment objects through video.

1

52

307

1

0

3

Jonathon Luiten

@JonathonLuiten

10 months

@janusch_patas @fdellaert Photo-SLAM and Colmap-free 3D Gaussian splatting.

1

0

3

Jonathon Luiten

@JonathonLuiten

1 year

@HelgeRhodin This looks like cool work!!! I wonder if there are things we could combine from your work to make the current Dynamic 3D Gaussians even better?

0

3

Jonathon Luiten

@JonathonLuiten

1 year

See these shapes actually rotating:

Jonathon Luiten

@JonathonLuiten

1 year

The core idea is enforcing that Gaussians have persistent color, opacity, and size over time; and regularizing Gaussians' motion and rotation with local-rigidity constraints. Dense 6-DOF tracking emerges from persistent dynamic view synthesis, without correspondence or flow input

1

51

0

1

3