🚨New NeRF app alert 🚨
Watch to the end for the results!
It’s called Captur3 and it’s already available on the
@Apple
App Store.
#3D
#AI
#neuralnetworks
Wow! Put yourself virtually in a real NBA game!!!
This is the
#augmentedreality
dream people have been waiting for! And the NBA is behind it, not just some small AR/VR shop.
#AR
#MixedReality
This is NOT a mesh, NOT a NeRF, NOT video footage. This is 3D Gaussian Splatting which produces a similar output to NeRFs. Check out those reflections in the water! The fine structure detail 🤩
#3D
#AI
#Computervision
#VFX
I wanted to model my hotel room this week and decided to give
@Polycam3D
's roomplan mode a whirl. 👏 Well done! They added a wonderful touch by putting 3D objects in the output that are actual representations of the room objects. Not just geometric shapes.
Why aren't more
Follow along as I do a series of structured
@Apple
RoomPlan tests and share my findings/notes in this thread.
First up, I tried tricking RoomPlan with a large mirror. Surprisingly it wasn't fooled! Also, it was way off on french doors height.
#WWDC22
#AR
#ARKit
#AI
@Scobleizer
How long before 360 camera tours are irrelevant? 2 years?
This is one of Zip-NeRF's 360 tours. I can look around, pause and recompose my shot. This is way better than a few static positions.
I only see 2 challenges today:
1. Quality - still not photorealistic, but I bet we will
Are kids, cars, and dogs messing up your PERFECT instant NeRF? Never again!
My upcoming tutorial will show you how you can automatically remove unwanted objects. Since it's automated, it only takes a few extra seconds to remove objects.
Thanks
@NVIDIAAIDev
for the enhancements!
FINALLY getting to try editing NeRFs in
#VR
.
This is a NeRF being rendered in real-time on my Oculus Pro Headset.
This one I already cleaned up a bit, but you can see me moving around the scene and removing some floaters.
Just a reminder, some floaters
🏎 3D Gaussian Splatting with cars is INSANE! 🏎
One aspect of NeRFs that has always plagued me is that car paint reflections looked blocky or smeared. Not here!!! Also, this is running in real-time.
@Scobleizer
👀
#AI
#3D
#Spatialcomputing
My take: NeRFs in game engines mixed with 3D model assets is going to be the key to long term NeRF success.
Here is one great example! NVIDIA will also be bringing NeRF interoperability to the Omniverse (my prediction).
#ue5
#gamedesign
⚠ The most important NeRF project I've seen since Zip-NeRF! ⚠
This team solved a plaguing problem that I run into: NeRFing long, linear, forward-facing image sets! Now I can NeRF alleyways and streets!
The trick here is 2-fold
1️⃣ Creating multiple-local radiance fields along
Made my first
@NVIDIAAIDev
NeRF with the
@insta360
RS One 1-Inch camera. I used 23 seconds of slow walking down a random alley as my test. Extracted 1 fps. I have some settings to tweak in my workflow to optimize the output. I did this with 6 lines in a CLI.
#AI
#NeuralNetworks
At the 11 second mark you realize what you are looking at 😏. What is your favorite aspect of the
@NVIDIAAIDev
Instant NeRF? I can't decide if I like the subtle cloud changes or water reflections more.
#neuralnetworks
#AI
#Drones
Scanned this telecom tower compound last week with an iPad Pro while with a client. Trying out point clouds with UE5 for the first time. This could go places...
#UE5
#Telecom
#industry40
I went a bit overboard scanning my brother's truck yesterday.
150 million points...from an iPhone. No data cleanup. I missed some of the top...but hey! I was trying NOT to slip on the icy snow!
You can achieve these the same results from the
@Pointprecise
app which is powered
Who’s ready for my world tour powered by
@NVIDIAAIDev
’s Instant NeRFs?!
The newest update is going to let me push this all a little further!
#InstantNeRFSweepstakes
#3D
(Fixed version of the cut off video)
I have never used Clips before in my iPhone or iPad. These AR affects are under appreciated! The room scanning experience is also fun…reminds me of scanning my room with the Magic Leap.
#AugmentedReality
#AR
Could NeRFs be used for ecological studies? I did not think all of the plants and foliage would turn out so good! The blurry/smeared bushes in the front and the mossy tree branch are only blurry because they were out of focus in in the source imagery.
@NVIDIAAIDev
#3D
#AI
Welcome to
@wooorldXR
on the new
@Meta
Quest Pro! I had sooooo much fun exploring in solo mode. THIS is how map navigation in VR should be done. Also, watch with sound on! 🔊
Thanks to the team at Wooooorld for letting me explore this app. Way more to come.
#MetaQuestPro
#VR
🚨 Let's clear up some confusion on NeRFs vs Photogrammetry 🚨
Matt Tancik - a leading NeRF researcher, explains in simple terms what a NeRF is, and more importantly, what it is not. We also talk about his work with
@nerfstudioteam
!
@Scobleizer
Haven't made a NeRF in a while. Today, I am going to update my "drag and drop" automation for the new Instant NeRF windows executable that
@NVIDIAAIDev
released. I will add automation for installing dependencies. Expect an update sometime today on my git:
No code Instant NeRF was too much work. I just posted a video on how to make a total end to end automated
@NVIDIAAIDev
Instant NeRF with your own images or video. Just drag a file or folder of images into a script then sit back and enjoy!
#AI
#Neuralworks
People are getting very excited about dynamic gaussian splatting, as am I. However, I would like to point out that samples like this one were recorded using 60 synchronous cameras. This will be a technology for studios with specialized setups.
#3D
#AI
The new
@Apple
Pay
#AR
transit experience is out of this world! I rode the subway to NYC and back from my backyard. I definitely know how easy Apple Pay will be for transit riders. Looking forward to more phenomenal
#augmentedreality
experiences like this in the near future.
My son no longer asks for his photo. He wants me to 3D model him instead. He moved his head slightly as I walked around. Other than that, this looks like another stunning output from
@LumaLabsAI
.
#ai
#NeuralNetworks
I love simulating movement in
@NVIDIAAIDev
Instant NeRFs. Check out these waves 😍 They are camera motion locked. I can fast forward or rewind the waves by moving the camera right or left.
#AI
#InstantNeRF
#NeuralNetworks
PSA: You're scanning everything WRONG!
Follow these three tips when capturing video or photos for
#photogrammetry
. That's right, this works for
@PhotoCatch3D
,
@scanamaze
,
@Polycam3D
or any photogrammetry software. Of course there are many other factors, but these are a must!
#3D
🎅 I almost missed
@NVIDIAAIDev
's Christmas present: No Code Instant NeRF!
That's right, they finally made a NeRF executable with no need to compile code!
In this video I walk you through everything you need to know.
#Computervision
#neuralnetworks
I had fun with this NeRF. The reflections in the windows are 😎 I also tested what would happen if I pushed in on the scene for the first 5 seconds of the source video...surprisingly it all came together with an interesting effect.
@NVIDIAAIDev
#3D
#architecture
#AI
I modeled my
@Nike
Metcon shoe using photos from the new
@apple
#iPadPro
and
#ObjectCapture
on my Macbook Pro. Here is the CRAZY part, it processed in less than 10 minutes. I will upload the model to
@Sketchfab
in a bit. There is one hole in the model from bad lighting.
So far,
@NVIDIAAIDev
Instant NeRF is winning the battle against
@LumaLabsAI
on linear unbounded captures. I think this is because they are taking 2 different approaches. On the left is Luma's pre-rendered video and me showing the interactive web views.
#3D
#AI
#NeuralNetworks
Extremely low light RoomPlan worked like MAGIC on my iPad Pro. You know what I’m doing in the morning! Scanning the whole house!
#WWDC22
#ar
#AugmentedReality
This is quite incredible! Perhaps one of my favorite NeRFs ever! I love the whole city as the camera quickly pulls away from the building.
@NVIDIAAIDev
#ai
#3d
I’m blown away from the results I am getting from then
#RayBanStories
3D scans using
@EveryPointIO
! I’m working on a more hands on video with some pros/cons of using
#smartglasses
. Should I do a TwitterSpace session to discuss with ya’ll?
#3Scanning
Blender AND Unreal Engine NeRF integration videos all in one day! Watch out everyone, NeRFs are going to inch into more and more creators' toolsets.
#NeuralNetworks
#AI
Just NeRFing a
@Ferrari
in case they want me to meet their marketing team somewhere to do a full 360 😎
Check out those reflections! The detail of the villa and gardens! Just so much to take in here! 150 images from a 1080 smartphone video.
#Ferrari
@NVIDIAAIDev
#3D
#AI
.
@NVIDIAAIDev
Instant NeRF from
@insta360
One RS 1-Inch camera!
Overall, it came out kinda "soft". I have ideas for improvement!
I decided to show the approximate imagery from source at the bottom right so you can see how much sharper the 360 footage is.
There was a nagging
BOOM! Instant wall, window, and door dimensions AS YOUR SCAN with
@Apple
's new RoomPlan API. This may not look pretty, we are just moving fast and seeing what works. If you look closely, it knows the difference between open and closed doors. Impressive!
#WWDC22
#architecture
#3D
Have you seen all of the amazing 3D Gaussian Splatting posts and want to try it out yourself! Well, I have a guide for you! This absolute beginners guide will walk you step by step to make your own scene.
#AI
#3D
#VFX
Quiz Time!
One is from Photogrammetry via a company’s photo mode. One is from NeRF via
@LumaLabsAI
’s mesh export. Which one is which? Leave a comment!
Also, how long before photogrammetry is replaced by NeRF tech?
@Scobleizer
@gravicle
I explored NeRFs several months ago for image stabilization. Worked well on static scenes. DynIBaR is an awesome implementation for dynamic scenes! WOW!
#computervision
#neuralnetworks
#VFX
Does this NeRF feel like a virtual Monet painting?
This scene would be TOUGH for photogrammetry to visualize.
NeRFs make it extremely easy to visualize scenes like this one that is full of flat, think structures.
What should I NeRF Next?
#AI
#neuralnetworks
#3D
#Art
I got 3D Gaussian Splatting for Real-Time Radiance Field Rendering up and running with my own data. Here is my quick first impression. Much more to come! Just follow the YT channel to see more as it comes out.
#AI
#3D
#NeuralNetworks
Perhaps my best
@NVIDIAAIDev
Instant NeRF yet! Those water reflections are stunning.
This would have been even better if I had a beefier GPU. I had to downsample images to stay within vram limits. The results are 🔥
#AI
#NueralNetwork
#Computervision
I've processed this kitchen dataset a TON of different ways into a NeRF. Let's take some guesses: what software did I use for this output:
A.
@NVIDIAAIDev
Instant-NeRF
B.
@nerfstudioteam
C.
@LumaLabsAI
Tomorrow I post the complete comparison breakdown.
This is the most images I have ever used in a NeRF: 330
I have a new GPU arriving with 2.4x capacity, bring on the images!
This NeRF is of the Memorial Union at
@oregonstate
Can you spot the robot in the video!?
#3d
#Computervision
#architecture
#AI
Watch till the 30 second mark, I added a little glow mode 🌟
It's been a while since I posted a NeRF. Life/work has been VERY busy, but I am excited about this one. I am going to use VR to manipulate this scene for an "art experience".
I love what
@NVIDIAAIDev
has been up to
I showed RoomPlan to a realtor. He jumped out of his bar stool! He said he wants this NOW to capture house layouts. He doesn't care about accuracy, he just wants his clients to understand layouts.
@Scobleizer
proves your point that Apple knows non tech people!
#realestate
I only took one video on my camping trip at Suttle Lake Lodge as we were headed out of the mountains.
I am quite surprised how well a 20 second iPhone video works for making an
@NVIDIAAIDev
#InstantNeRF
!
#3d
#architecture
#AI
THIS is one of the main uses of NeRFs - the accelerate the development of visualizing a scene.
❓ Are you excited about it?❓
Currently, my feelings about NeRF quality is that they are not quite game/or film production quality yet. However, for machine learning and simulation,
Introducing the Luma✨Unreal Engine alpha! Fully volumetric Luma NeRFs running realtime on Windows in UE 5 for incredible cinematic shots and experiences, starting today!
Try now:
I am working on a tutorial on "gamifying" a point cloud in
@unity3d
to make it navigable with collisions so you can move around in FPS Character mode. Here is a sneak peek of the navigation. Checked out my pinned tweet where I set up the stylized level using Stipple.
#Unity3d
I just completed a splats inspired DX12/OpenXR particle demo that runs on both AMD and NVIDIA, with my 4090 running it at a whopping 2x2224x2160 @ 120 FPS
I'm still processing the results, but the Quest 3/Vision Pro VR season is definitely going to be wild!
[1/n]
I'm going through all of my NeRF renderings to gather material for a presentation and I came across my Dolomites NeRF. This is one easily is one of my favorites. This was all the way back on June 1st when I made it.
@NVIDIAAIDev
#InstantNeRF
#Computervision
Pretty much getting my
@unitygames
VFX Graph education by combing through the setup that
@_kzr
created for bibcam vfx. I want to take this a whole new direction after this!
Tried Record3D to NeRF tonight. 1 minute capture, NO POSE ESTIMATION, trained for 4 minutes. Could have trained for longer. This was just a test.
Nice work
@nerfstudioteam
! This will be great for live demos! Next up, could I wifi stream this to NeRF training?
#3D
#AI
Take you 360 footage and use it for 3D Gaussian Splats and NeRFs!!!
This is the EASIEST but not the BEST way to use 360 imagery. Stay tuned, I want to collab with someone to make a far better video.
#AI
#3D
#VFX
#Computervision
I went overkill and used 1,283 images from a video to see this
@NVIDIAAIDev
Instant NeRF
What I learned: more images do not necessarily make the output better. However, it does help in areas with only one camera pass. 641 image version and more in the🧵
#AI
#NeuralNetworks
1/5
Trying out different NeRF techniques in the evenings. I am going to compare this NeRF with
@nvidiadeveloper
Instant NeRF,
@nerfstudioteam
, and
@LumaLabsAI
.
Who is going it win? Any bets?!
The result here is rendering in real-time in Instant-NeRF with an RTX 3090ti. DLSS enabled
My first NeRF using
#Nerfstudio
. Some aspects look way better than my Instant NeRF rendering, and some parts are a bit weird...like my legs doing their dissapearing trick. I learned a lot! Next video will be much better!
#AI
#3D
How does he have sub 200 followers? This NeRF is insane. He is inspiring me to do 360 NeRFs...but I literally shipped my pro 360 camera to a client last week. So do I get a new one to test?
߷ SPIn-NeRF: Multiview Segmentation and Perceptual Inpainting with NeRFs
I've posted about in-painting on NeRFs before. I find that this line of research will be highly useful for future applications.
Project page:
#VR
NeRF here were come!!!
Still LOTS to figure out. Stay tuned...and thanks for the tips
@gradeeterna
@Scobleizer
I am crossing every finger that this Fall I can make these on an
@Apple
Mac Studio or an updated Pro.
#NeuralNetworks
Bravo to
@Polycam3D
and
@cpheinrich
for being the first to launch Gaussian Splats commercially! My first test was of my cat on my desk using 170 images extracted from video. He moved a tiny bit and the white balance changed at once point, which you can see in the result.
#AI
#3d
Blender visualization of a flexible camera model I made for rendering in
#InstantNeRF
so you can see what’s going on. Who says we have to use perspective cameras all the time?! I love NeRF so much. Cheap and easy photorealistic renders is a game changer for digital surrealism.
A NeRF test set on the new
@NVIDIA
RTX 3090 ti.
It's not as crisp because I chose 24 fps to speed up rendering.
No hiccups on the upgrade and I can already tell the rendering is faster!
Tomorrow, I will try 1000 images in a NeRF!
#3D
#InstantNeRF
@NVIDIAAIDev
This is very cool! This project is being built on a NeRF Unity VR Project that I installed and played with about a year ago. Cool to see it being build on top of for industry.
This is really exciting!
@leo_nid_k
published a paper that shows it is possible to build 3D Guassian plat models in minutes on a CPU. He also produced meshes and more! Check out the link to the project in his post.
#AI
#3D
#Computervision
My newest paper: Flexible Techniques for Differentiable Rendering with 3D Gaussians
We show connections between 3D Gaussian renderers (FM & 3DGS), build fast models in minutes on CPU, optimize differentiable optical flow, and export colored meshes.
If you read one thig today, this is it. The article is interactive and teaches you a lot about cameras and lenses.
I am bookmarking this article for future reference.
Well this is interesting!!! Training my first
#nerfstudio
NeRF! I think I will have to make a quick tutorial for the community of people who aren't used to working with command line software.
#AI
#NeuralNetworks
🔥 Looking to make awesome camera animations with 3D Gaussian Splatting? Look no further! I made a tutorial! Thanks to the
@nerfstudioteam
viewer!
It's so easy, you can get up and running in about 15 minutes.
#AI
#3D
#VFX
NeRF compilation time ft. drones, phones and oh my! Photogrammetry ain't new, but it's never looked better thanks to AI. Plus, it's so satisfying to see it breath new life into old 3d scans! Watch the full 4 min video here:
#ComputerVision
#NeRF
#3D
#AI
Holy smokes it actually worked. Live preview of
@NVIDIAAIDev
#instantNeRF
rendering in Blender. This is the start of something very beautiful 🥹 and I can’t wait to share more of what I’m working on next.
Time for my kids to make this mess: 36 seconds
Time to make this NeRF and render video: 36 minutes
This isn't replacing Matterport tours
yet...
Capture method: 3 loops of 1080p video using iPhone 14 Pro ultra wide lens
Training Method: Nerfacto via
@nerfstudioteam
Next up,
Real-time photogrammetry on an iPhone is getting crazy! Our team RAN around a stockpile with the
@SRMeasure
app and it scanned the whole pile flawlessly!
He even got the measurement data in less time than it took to get his breathe back.
Here is the