We gave a tiny body to gpt-4o
Fully autonomous, give it just one prompt
Robot sees -> thinks -> does -> repeat
Can also:
• pick up trash
• shake hands
• talk with you
All for $200 off-the-shelf
Building a bigger version
What should we have it do?
@huggingface
's LeRobot team is awesome
Last week in Paris, their teleop was already up to 500Hz thanks to
@HaixuanT
's dora-rs. Feels very natural!
Great work
@RemiCadene
and team!
The year of robotics continues to 📈
Meet LeRobot, my first library at
@huggingface
robotics 🤗
The next step of AI development is its application to our physical world. Thus, we are building a community-driven effort around AI for robotics, and it's open to everyone!
Take a look at the code:
In the future AI Engineers won't replace us. Instead, we will orchestrate them!
And that's what we built this weekend at the
@MistralAI
@cerebral_valley
hackathon!
· Spin up devs or reviewers
· Define tasks for them on a board
· Check their code on Github
and more coming ⚡️
Given that this is blowing up, if you want to financially support an AI x robotics team building from Palo Alto, feel free to reach out
We're working hard on this
@emollick
Same impression, but from what I read on Reddit (), it might be because it's more "basic" by default, but more customizable to make custom GPTs
@Pranavi9228
Right now thinking of doing chores, once it gets better it will be able to help elders. But at first, laundry, tidying...
What do you think of it?
@JokesOnAI
Custom made from a kids car set to assemble, a raspberry pi, and a breadboard with all cables assembled myself
I was surprised this worked honestly
I actually built this Recall AI 2 months ago.
Runs entirely locally so that your data stays safe.
Can be used to figure out what you did during the day and how focused you were.
Should I release it?
Satya Nadella says Windows PCs will have a photographic memory feature called Recall that will remember and understand everything you do on your computer by taking constant screenshots
@TwitterDev
Will you deprecate the academic accesses before proposing new solutions for academics?
We have studies running over long timespans that would really suffer from an interruption
Want to try it for yourself?
The code and instructions are here:
Feel free to reach out for feedback or even to contribute!
We're already using this ourselves, and want to see what best works for the community!
@alvarocha2
Yes, it really behaves entirely autonomously
If you could see the computer screen there, the "thoughts" are coming in with what it decides on doing
Does not work all the time perfectly, but we're working on it. It's a complex but fascinating system
Today I saw the launch of another "open-source" AI wearable that has not published anything just to charge you 5x the cost
At
@MistralAI
x
@cerebral_valley
hackathon in
@SHACK15sf
we built FRIEND - an AI Wearable that:
- Works 24+ hours on a single charge
- Costs ~$20
-
This is the product of an incredible team who worked day and night to make this a reality:
@Aiswarya_Sankar
@yumat_10
and
@pierrecollignon
Hacking with amazing builders is what makes SF such an incredible city!
@Guygies
Could be a thing! But I think the best way to know what is going to be the best way to make a brain for these things is first to make the things. After making a couple of different ones maybe we'll be able to distribute a good software for people to use
In the future AI Engineers won't replace us. Instead, we will orchestrate them!
And that's what we built this weekend at the
@MistralAI
@cerebral_valley
hackathon!
· Spin up devs or reviewers
· Define tasks for them on a board
· Check their code on Github
and more coming ⚡️
Jensen Huang said engineers of the future will primarily be orchestrating a group of agents to build their visions.
Well that’s what we’ve built. Dropping our new OpenArchitect crew of engineering assistants from the
@MistralAI
SF hackathon this weekend in a few days! 👀
@KennethCassel
Working on a system without a depth cam here
Stereo cameras have a lot of issues (very disappointed by my OAK-D Lite), and ToF cameras are expensive - or very shitty.
Monocular Depth Estimation on the other end made huge progress. I'm hopeful
@emollick
Architecture changes, Better data, better performance given a fixed size, dedicated chips... And even fine-tuning for specific use-cases to reduce size further.
If each of these 10x's speed, we can easily get 5 orders of magnitude. Likely more over the years
@ClementDelangue
The world sort of needs open-source eval systems that you cannot train on.
Something that generates inputs / outputs to eval that are different every time?
But stay consistent in regards to what you're testing?
I actually built this Recall AI 2 months ago.
Runs entirely locally so that your data stays safe.
Can be used to figure out what you did during the day and how focused you were.
Should I release it?
Very, very impressive!
Can't help but notice the absence of legged locomotion while it's doing a task. Moving legs while you're doing a manipulation task in a clean way remains to be seen
Up next for real AGI: World modeling, long-term planning and seamless interaction
😫 How tiring is it to build a whole React frontend for every API you build?
Introducing...
🪄 Natural Frontend for
@FastAPI
: Generate a frontend for your backend with one-line.
👥 You can even make multiple frontends for different use-cases.
Completely open source!
@thedigitaldr
@MistralAI
@cerebral_valley
Not yet?
But you can host it yourself very easily (our repo indicates how)
You only need an api key from openai or mistral
We'll make it gradually easier for everyone to host it of course
@audrow
As a SWE/AI person, it really felt like there was so much legacy to learn to even make something work.
What do you think would help more people make robots faster? Software-wise?
@Thomasbcn
I remember Manifest V2 already made it much much harder to have my extension communicate between different domains... How more painful is this one going to be?
@xiao_ted
So true!
The funny thing is, if you make a co and ask VCs, they would probably tell you that they expect you to provide value now with something that works, not to rely on scale catching up later :)
@audrow
I would be happy to read about a page the lists the most important ones to learn about it
Cooking AI-based robots myself in my garage and this is an area I don't know anything about
In the future AI Engineers won't replace us. Instead, we will orchestrate them!
And that's what we built this weekend at the
@MistralAI
@cerebral_valley
hackathon!
· Spin up devs or reviewers
· Define tasks for them on a board
· Check their code on Github
and more coming ⚡️
@chris_j_paxton
@KennethCassel
Can't it be done in the cloud?
Depth estimation may not have to be done with such a low latency / high freq - ACT can do high freq-required task just with RGB.
Could also distill a local model to train on images seen and estimated with the cloud one after a while
@chris_j_paxton
It feels like they get the timing right though - essentially, Starship reducing the cost per kg launched by 1 OOM
By the time their proof of concept is done it should fly - if you believe everything will go right
@ai_jaksa
@eshear
The Bicameral Mind from Westworld
We're all LLMs talking to ourselves inside the brain in a Chain-of-Thoughts or Chain-of-Code fashion running at 1000Hz and our reflexes are lower-level transformers trained to follow any instructions from the high-level LLM
I actually built this Recall AI 2 months ago.
Runs entirely locally so that your data stays safe.
Can be used to figure out what you did during the day and how focused you were.
Should I release it?
I actually built this Recall AI 2 months ago.
Runs entirely locally so that your data stays safe.
Can be used to figure out what you did during the day and how focused you were.
Should I release it?
The problem is not detect spam bots online, it's to be sure you're not removing an actual human that just has a weird behavior
If you mess up even only once, that's censorship and it looks very bad
We should be less harsh with platforms' issues. Demanding, but not harsh
@audrow
@vishivishx
@abemurray
Yeah
Funny how at the end of the day it might not be who has the best agent or model that will win, just who interfaces it the best with humans
@KennethCassel
We built an agent-based robot navigating around and interacting autonomously without SLAM nor LIDAR, just a camera and a VLM. Works really well!
We used a ToF but now trying to see if it performs well without it and just DepthAnything
This is a correct way of comparing search engines and chatbots.
People complain LLMs hallucinate and thus are not accurate, but forget humans also make mistakes.
What's important is that LLMs are already faster and less error-prone than us (except if you take 15hr for search)
For any complex topic, the declining value of Google compared to AI-assisted search engines is getting clear.
Despite small issues, Bard, Bing & Perplexity do a much better job. I would also suspect that they are often less error-prone than "doing your own research" with Google.
In the future AI Engineers won't replace us. Instead, we will orchestrate them!
And that's what we built this weekend at the
@MistralAI
@cerebral_valley
hackathon!
· Spin up devs or reviewers
· Define tasks for them on a board
· Check their code on Github
and more coming ⚡️
@b05crypto
Not yet but it's on the roadmap!
Didn't have the time to cater to this part yet, we're mostly working on improving the coding agent because we want to deliver something that works well reliably :)
@VikParuchuri
Sweet! Can definitely see this being useful in may other projects. Maybe
@MikeBirdTech
useful for OI? I used surya already for some screenshot's text OCR
@nabeelqu
Is it really the whole of America? Is it not just parts of it? In SF it's easy to integrate but probably because most people are already from abroad
@ADarmouni
@mattshumer_
I think it only makes sense when you want your output to be formatted in a way that makes sense to your use-case or to get your model to get faster to the point
But fine-tuning it as a way to teach it new knowledge is pointless imo. Just do ICL for this
Idea for a human-computer interaction experiment:
Observe evolution of brain ECG while reading tweets and try to find correlations / predict reaction based on tweet
Reading a tweet is a bit like downloading an (attacker-controlled) executable that you instantly run on your brain. Each one elicits emotions, suggests knowledge, nudges world-view.
In the future it might feel surprising that we allowed direct, untrusted information to brain.
@analyticsaurabh
@cerebral_valley
@MistralAI
Thank you!
And yeah that's the goal. We don't think the future of AI is going to interact with chatbots
It's too limited
We can do much better than that
@traiden6
@huggingface
@HaixuanT
@RemiCadene
As a software guy, I am incredibly impressed by those who can make the hardware very fast to be honest.
AI / software is important, but if you're limited in your speed of iteration because you can't go fast on hardware, you're still f*cked
Built. Shipped. Delivered.
Now you can have full “architect” control over your crew of swe agents
- Describe what to build
- Discuss the best way to build it within your code
- Create tickets
- Agents build!
Thanks team and
@cerebral_valley
@SHACK15sf
best hackathon to date!
Are you managing comment sections or communities on a website?
@aliteshnizi
and I know it's hard, and we'd like to know more about your experience as part of our research at
@Stanford
on healthily engaging communities online.
Let us know if you're down for a chat :)
@eshear
The Bicameral Mind from Westworld
We're all LLMs talking to ourselves inside the brain in a Chain-of-Thoughts or Chain-of-Code fashion running at 1000Hz and our reflexes are lower-level transformers trained to follow any instructions from the high-level LLM
@s_streichsbier
Even without SOTA performance overall for small LLMs, we can probably reach incredible perf for some specific use-cases on devices. For a local github copilot, this one looks amazing
@onemoremichael
@ClementDelangue
Yeah, still game-able...
But at least not on the exact same labels of the original dataset so you still force the model to really understand the task at hand
@chichengcc
We should see that as a sign that hardware will learn to use very cheap hardware very well
Kind of akin to humans that are shaking can still do fine tasks because they learn how to compensate it
🤖🌎 I used the latest
@OpenAI
update to create a GPT for the planet in an hour.
✈️ It is made to help you travel more sustainably.
Just say where you want to go!
I don't think Twitter realizes the impact turning its
@TwitterAPI
off like that will have. It's useful for so many things, and research is one of them.
If you can at least grant academic access like before, that would be awesome for us researchers.