Ville Kuosmanen Profile Banner
Ville Kuosmanen Profile
Ville Kuosmanen

@VilleKuosmanen

624
Followers
227
Following
125
Media
491
Statuses

Founder of @voyagerobotics , building the future in public 🛳️

London, UK
Joined May 2020
Don't wanna be here? Send us removal request.
Pinned Tweet
@VilleKuosmanen
Ville Kuosmanen
1 month
Generalising to unseen objects. Mode was trained with coffee cups but can handle a translucent glass as well.
8
3
116
@VilleKuosmanen
Ville Kuosmanen
2 months
I bought a robot 🤖
Tweet media one
13
12
242
@VilleKuosmanen
Ville Kuosmanen
2 months
ARX arm running on @LeRobotHF teleop code 👀
2
20
186
@VilleKuosmanen
Ville Kuosmanen
1 month
Good to start the weekend with an improved model. Much smoother motions, and generalises to adding a second capsule to scene mid-way (zero-shot). Still too sensitive to capsule and cup placements in the scene.
3
17
148
@VilleKuosmanen
Ville Kuosmanen
1 month
Trained an ACT policy on the ARX arm, incredible seeing it complete a task successfully for the first time! 🦾
3
9
123
@VilleKuosmanen
Ville Kuosmanen
2 months
Took a while but arm teleop is working via ROS! Thanks @AgilexRobotics and @ARX_Zhang for helping out!
1
6
96
@VilleKuosmanen
Ville Kuosmanen
1 month
Assistant robot operator hard at work 🍊
Tweet media one
1
2
70
@VilleKuosmanen
Ville Kuosmanen
1 month
Replaying data collected via @LeRobotHF now works on the ARX5 arm. Want to pick the object? That’s what we need learned policies for! Hoping to get to that soon.
0
3
53
@VilleKuosmanen
Ville Kuosmanen
1 month
@JoshuaSteinman They did rebuild some of them! We watched an opera in the Odeon of Herodes Atticus in Athens last year! Highly recommend for the experience, enjoyed it even though I’m not into opera 🏛️
Tweet media one
Tweet media two
4
0
50
@VilleKuosmanen
Ville Kuosmanen
2 months
Managed to connect the AgileX base and cameras to Python (though latter still requires ROS). Movement is a bit inconsistent, need some way to determine how much it moved.
2
4
45
@VilleKuosmanen
Ville Kuosmanen
1 month
Robotics is cheaper today than ever. Remi's setup costs 50-100x less than what I started with - which is 10-100x less than what robotics startups had to pay just a few years ago!
@RemiCadene
Remi Cadene
1 month
"Robotics is traditionally dominated by big corporations and research institutions that have large budgets and resources, but the tutorials could support smaller players to get involved." Indeed! Thanks for the support 🙏
4
5
63
2
2
42
@VilleKuosmanen
Ville Kuosmanen
6 months
Managed to train an ACT policy for the Cobot Magic using simulation data that sometimes works. I need more, and more realistic, data but this is a good start!
3
3
33
@VilleKuosmanen
Ville Kuosmanen
1 month
One thing to consider when looking at e2e learning based robot demos: This is the “worst” shirt folding you’ll see a robot do. Unlike human-programmed robots, they will get better with every new data point we record.
@RemiCadene
Remi Cadene
1 month
First attempt at folding a shirt 😱 - Neural network predicts future motors position from camera inputs - Cameras of iPhone and Macbookpro - Robot arms cost 300$ each - Training over 100 examples takes half a day on Apple silicon Do it yourself with ⭐
38
176
1K
1
0
32
@VilleKuosmanen
Ville Kuosmanen
2 months
Installing experimental touch sensors, hope the wires don’t get tangled up
Tweet media one
1
0
26
@VilleKuosmanen
Ville Kuosmanen
28 days
Did a small co-training experiment, too lazy to type in a thread here so this screen cap from @LeRobotHF discord will have to do.
Tweet media one
2
1
22
@VilleKuosmanen
Ville Kuosmanen
2 months
Controlling the robot is not straightforward, only some of the joint controls work. Will have to reverse engineer the ROS1 based codebase 🫠
0
0
21
@VilleKuosmanen
Ville Kuosmanen
3 months
Put some more work on the RoboCasa integrations for @AgilexRobotics 's Aloha robot. Hoping to do some RL testing over the weekend. The binary 1 or 0 grip control doesn't work well, most manipulation tasks need granular control (tactile sensors might help as well👀)
2
2
17
@VilleKuosmanen
Ville Kuosmanen
4 months
Simulating a noisy GPS and compass sensor with a calculated camera pose. Simple mathematic SLAM stops working as the camera gets misaligned. Lots of research on how to fix this with SLAM techniques, most interested in neural methods using ML. Hoping to explore it in a few months
Tweet media one
1
0
16
@VilleKuosmanen
Ville Kuosmanen
5 months
Working on the @ai_habitat OVMM challenge is satisfying because it gives an excuse to finally fix all the minor issues with nav and perception that kill runs. Each may only improve success rate by 1% or so, but their impact compounds. Robot gets stuck much less often.
Tweet media one
2
3
13
@VilleKuosmanen
Ville Kuosmanen
4 months
stack now running successfully on Habitat Sim 3!
Tweet media one
1
1
13
@VilleKuosmanen
Ville Kuosmanen
3 months
going down the stack
Tweet media one
4
0
13
@VilleKuosmanen
Ville Kuosmanen
3 months
Quick observations on my experience so far from @ARIA_research Robot Dexterity program lead by @jcaread (1/n)
1
4
12
@VilleKuosmanen
Ville Kuosmanen
24 days
Thoughts on @notmahi 's Robot Utility Models paper: 1. data diversity really matters. I presume my problem of overfitted policies is caused by uniform data, and changing environments will allow the model to focus on the right things.
Tweet media one
2
1
12
@VilleKuosmanen
Ville Kuosmanen
5 months
Spent much of today trying to understand SLAM and how to calculate and track a robot’s camera pose across time and movement. Making progress but only got <100 lines of wip code after a morning and afternoon. It’s really hard!
2
1
10
@VilleKuosmanen
Ville Kuosmanen
3 months
The LeRobot paper presentations are one of the best ways to deep dive into cutting edge embodied AI research. Always full of signal, can’t recommend enough!
@RemiCadene
Remi Cadene
3 months
We are fortunate that inspiring researchers in the field are presenting their latest work in our reading group 🔥 It's taking place every 2 weeks 🤠 Join our discord for more info!
0
2
22
0
1
10
@VilleKuosmanen
Ville Kuosmanen
18 days
Cool new touch sensor just dropped 👀
@Raunaqmb
Raunaq Bhirangi
18 days
The sense of touch is fundamental to how we interact with the world. But the most exciting developments in robotics continue to focus primarily on vision. I spent the last four years trying to understand why. And we might have found a pretty good fix. Introducing AnySkin
11
65
442
0
0
10
@VilleKuosmanen
Ville Kuosmanen
9 months
Having played around with the @ai_habitat platform for the past two weeks I thought of making a small progress update. - built a simple web framework to display data broadcasted by the agent - experimented with using a Bayesian-inspired filter for image classification
Tweet media one
1
1
9
@VilleKuosmanen
Ville Kuosmanen
2 years
#Web3 authentication (with @MetaMask , @CoinbaseWallet and others) will put Auth0 and other centralised authenticators out of business.
27
0
0
@VilleKuosmanen
Ville Kuosmanen
2 months
I sometimes feel the goalposts moving on goals I’ve set > Build a prototype hardware device “Well, this one’s so primitive it doesn’t really count” Don’t fall to this trap, take time to celebrate the small wins and recognise how far you’ve come. Then set a new goal.
0
0
7
@VilleKuosmanen
Ville Kuosmanen
5 months
Co-training ACT with sim data from different robots (but same number of joints). I was expecting to need to make changes to model architecture but so far it looks to work pretty well out of box! Pink graph is baseline with data from a single robot.
Tweet media one
1
2
9
@VilleKuosmanen
Ville Kuosmanen
2 months
Uploaded a small dataset with touch sensors in the #LeRobot format: My old laptop can't really handle the volume of data the robot produces so the dataset is recorded at 2-5 FPS. Not ideal but we work with what we have 🤷‍♂️
2
1
8
@VilleKuosmanen
Ville Kuosmanen
2 months
figure showcasing a robot that creates real economic value
Tweet media one
1
0
8
@VilleKuosmanen
Ville Kuosmanen
7 months
@_akhaliq You wouldn’t steal an LLM
Tweet media one
0
0
7
@VilleKuosmanen
Ville Kuosmanen
24 days
@8teAPi We are also connecting models like these to robot bodies 🤖
0
0
8
@VilleKuosmanen
Ville Kuosmanen
4 months
Can robots understand the room structure of a house? Inspired by VLFM, this method evaluates images for different room types using an ITM model. 1st image detected a bedroom to the right, 2nd a kitchen at the top. Lots more work to do but looks promising!
Tweet media one
Tweet media two
1
0
8
@VilleKuosmanen
Ville Kuosmanen
3 months
If you are applying to the Robot Dexterity challenge and are interested in better sim integrations for tactile sensing (or tactile sensing in general), do not hesitate to contact me. DMs open! Keep up the great work @ARIA_research ! (7/7)
0
1
7
@VilleKuosmanen
Ville Kuosmanen
2 months
@subhransu_ro Around £30k. Not cheap.
0
0
7
@VilleKuosmanen
Ville Kuosmanen
1 month
Early employees get nothing with this type of acquisition. It reduces the incentive to work at a startup.
@KennethCassel
Kenneth Cassel
1 month
is this the new acquisition strategy since every acquisition by big tech gets blocked now?
Tweet media one
28
19
587
0
0
7
@VilleKuosmanen
Ville Kuosmanen
1 month
As a founder I know I'm supposed to Do Business (stonks) and not Do Research (not stonks) but I have seen a glimpse of what's possible and can't stop looking. Truth is calling and there is no one to tie me to a mast
0
0
7
@VilleKuosmanen
Ville Kuosmanen
4 months
One of my favourite DAOs using crypto to fund groundbreaking research into aging. Raising money for moonshot projects is never easy, so cool to see alternatives to grants & VC funding, plus anyone can get involved in the community. Fewer meme coins, more DeSci please 🙏
@vita_dao
VitaDAO 💛
4 months
24h left to join the VitaRNA Auction! 💛 VitaDAO is tokenizing the Artan Bio's IP-NFT, a groundbreaking gene therapy project led by biotech experts @Mykalt45 & @aschwartzphd 👉
0
6
21
0
1
7
@VilleKuosmanen
Ville Kuosmanen
5 months
Integrated the ALOHA wristcams into my agent in Habitat. Used for mapping but not object recognition (probably main cam is good enough for that). Not sure what the best UX is but the wrist cams looking down already make navigation significantly more reliable.
0
0
6
@VilleKuosmanen
Ville Kuosmanen
1 month
Robot in the video is fully autonomous, controlled by a pixels-to-actions transformer model trained with @LeRobotHF using 50 episodes for ~3 hours using an RTX 4090. Movement is def not perfect, more data and longer training periods will improve it.
0
1
6
@VilleKuosmanen
Ville Kuosmanen
2 months
testing data collection high quality data is essential in robotics so need to make sure frame rates and time alignment of different data sources is correct before recording real datasets
0
0
6
@VilleKuosmanen
Ville Kuosmanen
3 months
when you see this in the instructions you know you're in for a ride
Tweet media one
1
0
6
@VilleKuosmanen
Ville Kuosmanen
1 month
What people don’t see is that progress is first slow, then fast. Seeing a “toy” robot arm do a simple task autonomously is easily dismissed. But add 1000x more data and compute and only the sky is the limit!
1
0
6
@VilleKuosmanen
Ville Kuosmanen
4 months
Added a slight adjustment to room labels to make living room a less common classification. Result looks pretty accurate though still too noisy.
Tweet media one
0
0
6
@VilleKuosmanen
Ville Kuosmanen
1 year
there is no such thing as too much caffeine
0
0
5
@VilleKuosmanen
Ville Kuosmanen
2 months
I visited @imperialcollege on Friday - some incredible facilities for science and engineering there!
Tweet media one
0
0
5
@VilleKuosmanen
Ville Kuosmanen
1 month
Another day, another person telling me what I’m building is impossible and doomed to fail 🤷‍♂️ If people don’t tell you this you are not ambitious enough.
0
0
6
@VilleKuosmanen
Ville Kuosmanen
28 days
@chris_j_paxton I was surprised they only collected 26k examples. The paper described they had 35 workbenches and 8 months, it feels like you should be able to collect much more data with that setup.
1
0
6
@VilleKuosmanen
Ville Kuosmanen
5 months
Fixed the wrist camera issue (caused by render issues of close objects, you can see here as well) and changed top to front cam. I'd expect this to improve success rates for the policy. Time to test the hypothesis?
1
0
6
@VilleKuosmanen
Ville Kuosmanen
11 days
I think about this a lot. Stop writing AI slop, anon
Tweet media one
2
0
8
@VilleKuosmanen
Ville Kuosmanen
3 months
This is even more true for robotics. Cool demos and proofs of concept are hard but achievable. Building a robot that reliably does something of value is *so* much harder.
@yuris
Yuri Sagalov
3 months
@garrytan Many AI startups have an 80% problem. It's really easy to build an 80% solution. It's *really* hard to build a higher coverage solution. 80% will get you a POC, but you need better to win a contract. Very few startups have anything >80% right now — the field is wide open.
9
5
183
0
0
5
@VilleKuosmanen
Ville Kuosmanen
3 months
I'm building an open source tactile sensing framework for robots🤖! It will provide low-cost hardware designs, firmware, control software, ML policies, datasets, and pre-trained models. Free and MIT licensed! Give your robot a sense of touch with
0
0
5
@VilleKuosmanen
Ville Kuosmanen
3 months
I’ve already had productive discussions with other participants, and programs like this can build more than solutions - a UK ecosystem of successful robotics startups! (6/7)
1
0
5
@VilleKuosmanen
Ville Kuosmanen
2 months
We are hosting an embodied AI and robotics meetup in London! Join us for ⚡️ lightning talks ⚡️ and the drinks reception afterwards to meet fellow roboticists 🤖
1
0
5
@VilleKuosmanen
Ville Kuosmanen
7 months
Nice way to visualise detected objects by a robot. Hovering over the images shows the object detection mask. Built on top of @ai_habitat and other open-source code, next need to solve why the exploration algorithm tends to get the robot stuck in situations like this.
1
3
5
@VilleKuosmanen
Ville Kuosmanen
28 days
Everyone loves videos of robots doing stuff, so here they are. Planning to do a slightly cooler demo with these models later so 🙊
0
0
6
@VilleKuosmanen
Ville Kuosmanen
2 months
Set ambitious goals for yourself. It creates motivation out of thin air. Even if you fail, you probably achieved more than what you would have with a "realistic" goal.
0
0
5
@VilleKuosmanen
Ville Kuosmanen
6 months
Robot ordered, let's gooo! 🤖🚀
0
0
4
@VilleKuosmanen
Ville Kuosmanen
3 months
Ambitious ARIA is built for moonshots, and encourages you to think big. This is the energy we need - grants should aim to back risky and unexplored areas of research that won’t be backed by private investors. (2/n)
1
0
4
@VilleKuosmanen
Ville Kuosmanen
1 month
Is there robotics content on TikTok? Would people want to see short-form robot videos there? Never used the platform, someone please enlighten me.
1
0
4
@VilleKuosmanen
Ville Kuosmanen
1 month
Remi’s demo is trained with 100 examples - imagine an LLM trained with 100 pages of a book, or an image classifier trained with 100 images. We haven’t even scratched the surface of what robots can learn. Scaling these datasets 1000-1 million times will change the world.
1
0
4
@VilleKuosmanen
Ville Kuosmanen
6 months
@Nowooski @Noahpinion I feel the London job market is really two markets - one for globally competitive industries, often in finance, where pay is not far from US levels, and one for everyone else which is what this post references.
0
0
4
@VilleKuosmanen
Ville Kuosmanen
3 months
Trained the same ACT policy (simulated data) with (green) and without (blue) tactile data. No major differences between the two. Possible reasons: - task may be too easy - scripted policy (data source) does not use tactile feedback - 3 contact points per finger may not be enough
Tweet media one
2
0
4
@VilleKuosmanen
Ville Kuosmanen
1 month
But not everything works out of the box. Alternate target placements are challenging for the model, as is picking a fallen-over capsule (but pretty cool how the robot retried a few times, this is emergent behaviour!) I’m confident more data would fix these issues.
0
0
4
@VilleKuosmanen
Ville Kuosmanen
9 months
That Tesla demo makes me think whether a robot foundational model just requires thousands of hours of data collected of humans doing tasks while wearing VR glasses and some kinds of hand motion sensors.
2
0
4
@VilleKuosmanen
Ville Kuosmanen
2 months
cooking👀🍳
Tweet media one
0
0
4
@VilleKuosmanen
Ville Kuosmanen
2 years
Centralised providers are expensive and complicated to integrate, and require a round-trip to an external sign-in page. I never really got Auth0 integration working. Meanwhile, it took me less than a day to integrate @MetaMask into an app.
5
0
0
@VilleKuosmanen
Ville Kuosmanen
5 months
Got a basic teleop version working, but it's super laggy and hard to control. Lag may be something I just have to work with, but think the robot-side control processor can be improved to make the controls more fine-grained.
1
0
3
@VilleKuosmanen
Ville Kuosmanen
3 months
waiting for training to complete
0
0
3
@VilleKuosmanen
Ville Kuosmanen
6 months
Give your support agent access to tools. This would work much better with GPT-4 but the small Mistral model is surprisingly capable as well, if you don't mind the occasional mistakes.
0
0
2
@VilleKuosmanen
Ville Kuosmanen
10 months
Trying to generate illustrations with the same consistent look and feel to them using the new ChatGPT x Dall-E integration reminds me why graphical designers and illustrators won't go out of business any time soon.
2
0
3
@VilleKuosmanen
Ville Kuosmanen
3 months
you can just build things. no need to ask for permission
1
0
3
@VilleKuosmanen
Ville Kuosmanen
5 months
close enough to pass? (task was to move a tomato from table to sink. new heuristic for calculating candidate instances works surprisingly well first time!)
Tweet media one
1
0
2
@VilleKuosmanen
Ville Kuosmanen
28 days
@chris_j_paxton Looks like data quality is pretty important with these models. Feels different from LLMs that can learn from internet scale data with lots of rubbish. Wonder if it will flip at some point and even lower-quality data will give marginal improvements?
0
0
3
@VilleKuosmanen
Ville Kuosmanen
5 months
it only takes one successful run to go from "it's so over" to "we're so back"
1
0
3
@VilleKuosmanen
Ville Kuosmanen
29 days
@chris_j_paxton Both RR and Covariant seemed to target large enterprises that already have robots. I think the best approach is to target small businesses that don’t have robots yet, and make integrating them into operations as easy as possible.
0
0
3
@VilleKuosmanen
Ville Kuosmanen
2 years
@MetaMask - No need to fill in usernames. You wallet address is the username - Passwordless by default. Just prove your ownership of the account by signing a text sent by the back-end. Can't leak passwords if there are no passwords!
8
0
0
@VilleKuosmanen
Ville Kuosmanen
24 days
Happy to see this featured by @sotalikesfuture 😃🤖
@sotalikesfuture
Society for Technological Advancement
24 days
@joseflchen @climate_ben @gruffyddgozali 5/ I used a vision-language model to map a flat – a letter from @VilleKuosmanen
Tweet media one
1
0
6
0
0
3
@VilleKuosmanen
Ville Kuosmanen
6 months
Is this why the US 🇺🇸 economy seems to have more momentum than ever, while the UK 🇬🇧 and EU 🇪🇺 lag behind?
@JWMason1
JW Mason
6 months
It's hard for me to understand how anyone looks at the experience of the past few years and concludes that the kind of industrial policy being pursued by the US (call it "derisking" if you like) has not been effective.
Tweet media one
14
60
201
0
0
3
@VilleKuosmanen
Ville Kuosmanen
11 months
@Noahpinion Possibly Finland over Russia depending on how you define colonisation, but not over Sweden. *Russian rule was more like occupation than colonisation, and the same can be said about Poland as well.
1
0
3
@VilleKuosmanen
Ville Kuosmanen
1 month
@waitin4agi_ Many rent seekers will do their best to convince you they are builders too.
0
1
3
@VilleKuosmanen
Ville Kuosmanen
1 month
A major takeaway from @chris_j_paxton 's OVMM challenges was the difficulty in recovering from bad navigation or manipulation actions - robots often got stuck. Learning-based robotics is exciting because it seems to give a Get Out of Jail Free card when things go wrong!
@AtharvaPusalkar
Atharva Pusalkar
1 month
Recovery behaviors using diffusion policies: Here’s a short demo in which the robot loses its grip on the handle and then quickly recovers by pushing the door open. @chris_j_paxton @hellorobotinc
3
16
78
0
0
3
@VilleKuosmanen
Ville Kuosmanen
3 months
If your pricing page says “contact us”, I already know it’s overpriced. If you were confident your product is good value for money, you’d publish the price for everyone (including competitors).
0
0
3
@VilleKuosmanen
Ville Kuosmanen
3 months
Jenny has been incredibly helpful with feedback and bouncing off ideas. This structure ensures the best projects get funded, not just the ones that can navigate a difficult and bureaucratic process. (4/n)
1
0
3
@VilleKuosmanen
Ville Kuosmanen
2 months
If you'd like to do a 5-10 minute informal talk on anything robotics and embodied AI related, do contact me! DMs open, great opportunity to join in the conversation.
0
0
3
@VilleKuosmanen
Ville Kuosmanen
4 months
My local VLM was running slow - turns out there was an error with the llama.cpp model quantisation. After a rerun, the full model loads into CUDA and answers pretty much instantly. Thanks to @asoare159 for noticing!
0
0
3
@VilleKuosmanen
Ville Kuosmanen
5 months
Just had my phone correct “teleoperation” to “teleportation” while raising TODO items. I’m all for accelerating technological progress but maybe hold your horses on that 🐴
0
0
3
@VilleKuosmanen
Ville Kuosmanen
6 months
experimenting with mujoco this afternoon
0
0
3
@VilleKuosmanen
Ville Kuosmanen
3 months
think I’m the one on the left
@jonnyclifford
Jonny Clifford
3 months
Maddening how many people think that not doing a startup is best way to get better at doing a startup
Tweet media one
1
3
48
1
0
3
@VilleKuosmanen
Ville Kuosmanen
1 month
@kscottz The operational aspect of hiring data collectors, finding clients to outsource work, cleaning and managing datasets etc. is important and definitely not a trivial problem. But I don’t like how they presented an open-source tool as their own work and gave no credit to inventors!
0
0
3
@VilleKuosmanen
Ville Kuosmanen
25 days
Robot doing his best packing shoes - not fully autonomous yet (had to make a small correction) but getting there! @adidas @Nike would you hire him?
1
0
4
@VilleKuosmanen
Ville Kuosmanen
5 months
Example of a planning failure - robot could not fit through gap to the right, and planning a route all the way to the left failed. Could be solved by adding waypoints to explored space and treating them as a graph, but setting the viewpoints up feels very difficult in practice.
Tweet media one
1
0
3
@VilleKuosmanen
Ville Kuosmanen
10 months
@chris_j_paxton Hey Chris, as someone who took a crack at the challenge as an independent researcher, I can offer a slightly different take: If you don’t have access to large amounts of GPU compute, it’s very difficult to experiment with learning based strategies, let alone train them.
2
0
2
@VilleKuosmanen
Ville Kuosmanen
3 months
Starting with simulated touch sensor tests: using <site> with non-standard shaped geoms causes flickering. thinking of adding small elevated "bumps" that represent the sensors, to better capture the collisions (this is kind of realistic as well!)
1
0
2
@VilleKuosmanen
Ville Kuosmanen
6 months
it's just not fast enough
Tweet media one
0
0
2
@VilleKuosmanen
Ville Kuosmanen
5 months
Another small feature to help unstuck a robot - the red dot represents a position where a movement action failed. It adds extra cost to navigation planning, discouraging navigating through it but not preventing it.
Tweet media one
2
0
2
@VilleKuosmanen
Ville Kuosmanen
5 months
@chris_j_paxton I actually switched to the Spot agent planners (RRTConnect + other bits, great work btw!) + bunch of small changes on top. The base version works pretty well in like 90% of cases but has a habit of getting stuck. had to put a lot of work into error correction and retries.
1
0
2