Lucky Iyinbor @Luckyballa Twitter profile

Pinned Tweet

Lucky Iyinbor

@Luckyballa

4 months

Found some time to add scene collisions to my Vision Pro playground, and I did not expect this level of accuracy!

205

1K

14K

Last Seen Profiles

@StwGendut

@penyukastw21

@Gazal_horny

@NduMncube

@OBubble47080

@DougleisysM

@StalkingVi96051

@intelCDL

@mohdirfan1111

@Goat20475488

@Billy75Williams

@malaladias

@tabetha85642

@BinorRaja

@munakata2023

@thesplatt

@StwGendut

@holbysiain

@VartanYuji20302

@CinthiaReg26320

@bokeplokalmalam

@CLRI_LTC

@daznguyen98

@roma2023_24

@MattherTer95017

@Bull181146

@AdrienneLambCBC

@bokeplokalmalam

@lunvlr

@HAL1053595

@jacklynn4451

@dul_turkporno

@kubi_movie

@GSilva68460

@BinorRaja

@JanetteJud55725

Lucky Iyinbor

@Luckyballa

5 months

I migrated my experiment to the brand new LowLevelMesh API and fixed the pinch gesture, feels good! One benefit of using this API instead of hacks is that shadow casting now works properly, adding a great deal of realism ↓

47

161

2K

Lucky Iyinbor

@Luckyballa

4 months

Many have asked about the API enabling this level of interaction — it's called SceneReconstructionProvider.

12

103

1K

Lucky Iyinbor

@Luckyballa

3 months

I'm happy to share my new open-source Vision Pro example project called «Photons». It's a particle simulation demo that literally gives you superpowers!

28

81

943

Lucky Iyinbor

@Luckyballa

4 months

My dog is probably tired of seeing me waving my hands and pinching the air 😁

8

2

515

Lucky Iyinbor

@Luckyballa

3 months

It took way too much time to publish the project, all because I spent 20% coding and 80% having fun 😂

15

25

456

Lucky Iyinbor

@Luckyballa

2 months

SDF to Mesh in fragment shader 🫵

4

30

345

Lucky Iyinbor

@Luckyballa

2 months

Destructible SDF ⛓️‍💥

4

27

301

Lucky Iyinbor

@Luckyballa

3 months

Tetrahedral Skinning ✔️ Neo-Hookean XPBD ✔️

6

28

289

Lucky Iyinbor

@Luckyballa

3 months

Mesh Mortal Kombat: Real-time voxelized soft-body destruction — 2024 SIGGRAPH real-time live winner, well deserved 🦾

5

24

198

Lucky Iyinbor

@Luckyballa

2 months

I did some research on mesh extraction from SDF last weekend, and I didn’t like what I saw… The best techniques I was able to find were dual contouring, surface nets, and restricted Voronoi adaptation, which only works for dense grids. Is there anything better?

10

174

Lucky Iyinbor

@Luckyballa

3 months

A very interesting technique for mesh simplification and skinning weight optimization. While it may seem trivial, extracting a single layer from a multi-layered object while avoiding collisions is far from easy.

2

34

167

Lucky Iyinbor

@Luckyballa

4 months

@LinusEkenstam Thanks! It’s still early days for Vision Pro, and there aren’t many use cases yet, but it’s the most impressive tech I’ve ever used.

0

1

112

Lucky Iyinbor

@Luckyballa

3 months

Here is a public TestFlight link for the cloth demo, have fun!

TestFlight - Apple

Using TestFlight is a great way to help developers test beta versions of their apps.

testflight.apple.com

7

14

108

Lucky Iyinbor

@Luckyballa

2 months

2

5

107

Lucky Iyinbor

@Luckyballa

2 months

The love letter to Voronoi 💔 I used bisector cell clipping by @BrunoLevy01 and the team to identify Voronoi vertices. It’s a very elegant technique! Super useful for triangulation, RVD, and other things.

4

5

104

Lucky Iyinbor

@Luckyballa

3 months

VR-GS: A Physical Dynamics-Aware Interactive Gaussian Splatting System in Virtual Reality Very similar idea to tet proxy mesh, but for GS with some two-level embedding tricks to avoid rendering artifacts. It’s not mobile VR though, but it can be done.

2

15

103

Lucky Iyinbor

@Luckyballa

3 months

Connecting Voronoi neighbors by verifying that the voxel at the circumcenter contains all vertices of the potential primitive results in an almost perfect Delaunay tetrahedron, just as in the 2D case. I did almost no CPU calculations, but it feels like I am pretty close 🙂‍↔️

4

7

99

Lucky Iyinbor

@Luckyballa

5 months

I decided to spend time reimplementing my broad-phase collision detection, and it paid off! Instead of heavily tuned voronoi particle tracking, I am now using GPU spatial hashing. Now I can do 40k vertices (85k triangles) in under 15ms, getting closer to the goal🫡

5

4

90

Lucky Iyinbor

@Luckyballa

6 months

Here we go, I finally implemented VBD with the StVK hyperelastic model (λ = 1e5, μ = 5e4). Currently, I do: - 8 (max) force elements per vertex in parallel - 4 vertices in an SIMD group (8*4) - 0 loops for reduction - A lot of loops for Hessian 😅

3

1

91

Lucky Iyinbor

@Luckyballa

3 months

It looks like I'm ready to simulate the proxy mesh. Here is a discrete Voronoi with quasi-random initialization, weighted by mean curvature, and bounded by SDF. It uses constrained Lloyd's for relaxation and builds Delaunay tetrahedrons, which I render with ray marching.

2

4

88

Lucky Iyinbor

@Luckyballa

2 months

Delaunay 🤝 Voronoi

3

5

85

Lucky Iyinbor

@Luckyballa

3 months

Honestly, I'm not sure how I ended up here, but here's what I'm experiencing right now 🫠

5

3

80

Lucky Iyinbor

@Luckyballa

5 months

Optimizing for robustness over accuracy can get you pretty good results! Now I am doing soft vertex-vertex collision after initialization and hard vertex-triangle collisions during the solve step. Here is a stress test with pretty high velocity and elasticity —

5

6

75

Lucky Iyinbor

@Luckyballa

3 months

Took a step back and added surface reconstruction. Instead of Lloyd’s relaxation, I use inverse cube law energy, which I minimize with VBD. To keep vertices on a surface, I use closest point attraction. I will optimize it jointly with internal tets with almost zero overhead.

1

3

64

Lucky Iyinbor

@Luckyballa

4 months

@CameronR_S3XY Yes, it is! M2 is quite powerful

0

59

Lucky Iyinbor

@Luckyballa

5 months

It's so much fun to play with! I have a few ideas for improving the pinch gesture, but it already feels nice 🤤

1

8

57

Lucky Iyinbor

@Luckyballa

3 months

TestFlight link is finally here —

Join the Spatial Photo(ns) beta

Available on iOS

testflight.apple.com

Lucky Iyinbor

@Luckyballa

3 months

It took way too much time to publish the project, all because I spent 20% coding and 80% having fun 😂

15

25

456

5

8

56

Lucky Iyinbor

@Luckyballa

3 months

Well, that’s something! At least I‘ve implemented an SDF renderer that will help me debug my attempts to build tetrahedral meshing. All I need is some nice density function…

1

4

52

Lucky Iyinbor

@Luckyballa

5 months

Proper collision handling is very satisfying to observe, so I chose it next! This shows a collision with friction as described in the paper (using IPC). No self-collisions yet, but I can already see the improvement over other methods 🤩

5

3

46

Lucky Iyinbor

@Luckyballa

3 months

It’s powered by «simulation-tools» package – my open-source project focused on developing GPU algorithms for simulations and related tasks. Links:

GitHub - computer-graphics-tools/simulation-tools

Contribute to computer-graphics-tools/simulation-tools development by creating an account on GitHub.

github.com

4

47

Lucky Iyinbor

@Luckyballa

3 months

It is still in a very experimental stage, but if anyone else wants to play with it, let me know, and I will add you to the TestFlight. In the meantime, I am cooking something else 👨🏾‍🍳

9

0

46

Lucky Iyinbor

@Luckyballa

5 months

#WWDC24 magic is real 🪄

2

5

41

Lucky Iyinbor

@Luckyballa

5 months

If you are curious about building complex physics for Vision Pro or any Apple device, the best way is to start by learning Metal Compute. It’s simpler than you think! Here are some great tools and materials by my friend @eugenebokhan →

eugenebokhan - Overview

A software engineer with experience working with iOS, CG, CV, and AR. - eugenebokhan

github.com

2

4

39

Lucky Iyinbor

@Luckyballa

28 days

EdgeRunner: Auto-regressive Auto-encoder for Artistic Mesh Generation This paper introduces a tokenizer based on the EdgeBreaker algorithm to sequentially traverse triangular meshes. It’s great to see a classic CG algorithm from the last century being used to push the state of

2

9

43

Lucky Iyinbor

@Luckyballa

3 months

Okay, the idea of building a soft body tool for visionOS is growing on me quickly… Building a simulator only can be quite boring, so here is my plan ↓

Lucky Iyinbor

@Luckyballa

3 months

What’s next? 🤔

1

0

5

1

37

Lucky Iyinbor

@Luckyballa

4 months

@sampsonslayer Thanks! Yes, It works both with virtual and real objects, as long as real objects are reconstructed by Vision Pro scene API

2

0

37

Lucky Iyinbor

@Luckyballa

3 months

I need to constrain the tetrahedralization process to avoid out-of-shape connections, but it’s not clear how to do it robustly. I can’t rely on SDF - some edges could be outside. I tried to fill empty space with indicator seeds, but it didn’t go well. I am open to ideas!

7

3

34

Lucky Iyinbor

@Luckyballa

11 months

Here's a more technical showcase: - 24,000 vertices - An XPBD solver - Self-collision - Mesh collision - Multiple constraints - Various materials (per vertex configuration) All in real-time, starting from iPhone 12

2

3

33

Lucky Iyinbor

@Luckyballa

2 months

This is a result of surface nets vertex initialization and energy-based relaxation with Voronoi-Delaunay triangle extraction. No CPU post-processing, so there are holes and imperfections. Next, I will try to do Voronoi edge - SDF intersection test to improve connectivity.

1

2

32

Lucky Iyinbor

@Luckyballa

5 months

Funny enough, the performance on the iPhone 15 Pro is roughly the same as on the M1 Ultra Mac Studio 😅 Obviously, the iPhone only lasts a couple of minutes before thermal throttling kicks in

4

1

28

Lucky Iyinbor

@Luckyballa

3 months

My attempt to explore Gaussian and Mean curvatures for SDF

3

5

27

Lucky Iyinbor

@Luckyballa

2 months

It’s time to start thinking about what I am building it for.

1

0

28

Lucky Iyinbor

@Luckyballa

5 months

Here, I implemented simple bending and ported my XPBD self-collision implementation by converting constraints to an energy-based approach. To my surprise, even without conversion, we can run constraint-based resolution as post-processing to initialization, and it will still

2

0

24

Lucky Iyinbor

@Luckyballa

11 months

I am back on twitter! Check out the latest article about our efforts on cloth simulation at @zero10_ar

80 LEVEL

@80Level

11 months

. @zero10_ar has shared a comprehensive breakdown discussing the evolution of cloth dynamics and explaining how they approach cloth simulation for AR. Read the breakdown: #3dart #3d #3dsimulation #clothsimulation #ar #augmentedreality #clothsim #3dsim

4

24

160

1

2

26

Lucky Iyinbor

@Luckyballa

2 months

I am considering pursuing a part-time PhD in fields such as discrete differential geometry or differentiable physics. Is it worth it? Are there any benefits beyond career and collaboration opportunities in research?

6

0

25

Lucky Iyinbor

@Luckyballa

3 months

A few things I want someone to try: •Add ambient occlusion (could be done analytically) •Implement SPH for fluid simulation •Render particles as liquid •Use a mesh as an initial shape •Add shadow maps

Lucky Iyinbor

@Luckyballa

3 months

I'm happy to share my new open-source Vision Pro example project called «Photons». It's a particle simulation demo that literally gives you superpowers!

28

81

943

0

22

Lucky Iyinbor

@Luckyballa

5 months

One thing I particularly enjoy about vertex-triangle collision handling is that it allows to resolve penetrations using surface information. Applying cosine similarity tests against the vertex-triangle pair in the rest state significantly enhances DCD for multilayered garments.

0

4

22

Lucky Iyinbor

@Luckyballa

3 months

I do tetrahedral skinning similarly to the amazing example by @matthimf But instead of a hash table, I do JFA, so we always know the closest circumcenter, and it’s much faster on a GPU.

5

1

23

Lucky Iyinbor

@Luckyballa

5 months

Slightly enhanced my broad phase collision detection with some tricks for better reuse of cached proximity pairs. Collisions still make up ~40% of the pipeline, but I hope to reduce the cost further with CCD, allowing larger steps.

1

0

22

Lucky Iyinbor

@Luckyballa

4 months

It explicitly provides developers with scene mesh vertices and indices. How you use them is up to you. You can go beyond collisions, doing things like occlusions, custom shadows, drawing, and more.

1

0

22

Lucky Iyinbor

@Luckyballa

4 months

Link →

SceneReconstructionProvider | Apple Developer Documentation

A source of live data about the shape of a person’s surroundings.

developer.apple.com

0

2

22

Lucky Iyinbor

@Luckyballa

3 months

Physics! Started with simple XPBD edge and volume constraints. Next up: VBD + Neo-Hookean. Wish me luck…

1

3

20

Lucky Iyinbor

@Luckyballa

5 months

Lol, I also got it running on VisionPro with sync between SceneKit for the viewport and RealityKit for AR, plus added some hand interactions 🤌🏾

1

2

19

Lucky Iyinbor

@Luckyballa

4 months

@Charles_2M Good question! I don’t know yet, but I bet such scene understanding could be useful for AR games, immersive experience and etc.

3

0

19

Lucky Iyinbor

@Luckyballa

4 months

@XiexeVR It’s collision with a mesh, but sure, depth can be extracted too

1

0

18

Lucky Iyinbor

@Luckyballa

4 months

I am in the process of interview prep, so I am spending time grinding LeetCode instead of prototyping more cool stuff. It's a weird reality, but I accept the challenge 🫡

1

0

18

Lucky Iyinbor

@Luckyballa

3 months

I took a side quest and decided to implement a heat method for Generalized Signed Distance in Shadertoy, which brought me back to the X😅 The paper:

1

4

17

Lucky Iyinbor

@Luckyballa

10 months

One of most recent rabbit holes I went down was about geometry processing, specifically remeshing algorithms. I spent quite some time on it, but I am happy with what I got and learned ↓🧵↓

2

1

17

Lucky Iyinbor

@Luckyballa

2 months

SDF-RVD looks pretty though, I will try it out. The algorithm would be: 1. Initialize seeds 2. Complete 3D Voronoi 3. Project seeds on SDF 4. Get 3D Voronoi edge intersection points with SDF to build “2D” cell 5. Compute centroids to update seeds 6. Repeat

0

1

17

Lucky Iyinbor

@Luckyballa

5 months

I spent an eternity trying to figure out how to properly update vertex positions in RealityKit, so I just have to share a quick guide now. 1. Compute your positions. 2. Create a shader that takes the positions as a buffer and stores them in a texture. 3. Create a DrawableQueue

3

1

16

Lucky Iyinbor

@Luckyballa

6 months

Quick update on my VBD performance improvements: - Improved graph coloring with MCS algorithm: fewer, more balanced colors for better GPU utilization - Updated local solver to use Cholesky decomposition: faster on Apple’s GPU - Bug fixes and improvements for mass-spring kernel:

3

1

16

Lucky Iyinbor

@Luckyballa

3 months

This field of research is promising, but "classic" simulators are still undefeated. While HOOD has demonstrated an excellent use case for GNN and energy-based loss, it's not efficient or accurate enough to replace non-neural methods. This new paper is a great step forward!

Michael Black

@Michael_J_Black

3 months

Tick tock... 2:20 today at #SIGGRAPH2024 , I'll present "ContourCraft: Learning to Resolve Intersections in Neural Multi-garment Simulations." Mile High 4. @ArturGrigorev57 , the lead author didn't get a visa so I'm the designated hitter. What's it about? 🧵

3

13

105

2

0

16

Lucky Iyinbor

@Luckyballa

5 months

A new RealityKit API for dynamic mesh updates is here! Seems like I can throw away my weekend hacks and just put positions directly in the vertex buffer from the GPU.

LowLevelMesh | Apple Developer Documentation

A container for vertex data that you can use to create and update meshes using your own format.

developer.apple.com

1

2

15

Lucky Iyinbor

@Luckyballa

5 months

Here is my attempt at implementing the Two-Way Continuous Collision Handling paper, but in 2D 👾 It seems to be working, but if you are into simulation or math, please check the code against the equations for me 🥹

1

2

13

Lucky Iyinbor

@Luckyballa

2 months

Mesh to SDF using JFA is super fast (1000+ fps), but there is a known issue with a sign which can lead to artifacts. One way to fix it is with ray tracing, but it requires costly BVH construction. Is there any known method that is as fast, but with sign guarantees?

3

1

13

Lucky Iyinbor

@Luckyballa

4 months

@init_bobjames Xcode + Swift + Metal + ARKit + RealityKit + SwiftUI

3

0

12

Lucky Iyinbor

@Luckyballa

3 months

You know what's really mind-blowing? The Metal debugger in Xcode beta doesn't crash! I've used it multiple times for both macOS and visionOS apps, and it just works 🫨

1

0

12

Lucky Iyinbor

@Luckyballa

3 months

I missed the opportunity opportunity to call it Spatial Photo(ns) 🥲

0

11

Lucky Iyinbor

@Luckyballa

2 months