My 4th appearance on a podcast! I'm becoming a better guest (interviewee?) each time, which means this one is the best one :)
This time we cover colocated queries, materialized views, sqlite & crdts.
I never thought cross-device sync and multiplayer could be sooooo easy.
With CRDTs on SQLite I just declare which tables are synced and voila, my app is multiplayer.
One of the things I'm most excited for is partnering with the LibSQL folks to create a custom syntax for defining tables which are backed by CRDTs!
LibSQL is truly getting SQLite ready for the coming decades --
Interested (or know someone interested) in databases? Conflict-free data types? p2p?
Looking for help on
- ~130 companies signed up for the beta release
- $16,000 in paid sponsorships over the last 6 months
- 1.2k github stars
- Front page HN 3x
"Most databases were built when data was static and queries were dynamic. For applications, most queries are static and the data is dynamic" -
@matthew_linkous
I never thought about it this way before but this is spot on.
Working on a post but I present a React table of 2 million rows, dynamically filtered in real time.
tldr: the data structure backing the table is maintained via differential dataflow so we only need to process the single row that changed rather than the entire set when filtering
I was blown away by
@aboodman
's talk and the Replicache model so I spent the last day experimenting with a variation on the idea:
"Creating CRDTs without specialized knowledge"
This confluence of things:
- Event sourcing
- CRDTs & Causal Graphs
- Incremental View Maintenance
is going to unlock some stellar new infrastructure.
1. Event source your entire app.
2. Incrementally maintain app state as a view of events.
3. Rebase events for multiplayer.
Next.js and React have never been easier, and they've also never been harder ๐งต
Here's the universal 'minimum viable app', a TODO list, implemented with the latest features in 45LOC
What a clever way to automatically run a destructor when a variable goes out of scope in JavaScript. A generator that yields and has a `finally` block. ๐คฏ
I realize I never really talk about the developer experience of having SQLite in the browser + sync.
Working on a series of 10 minute videos to cover this.
Part 1 Setting up SQLite in the browser: solving persistence, shared state & cross-tab sync:
In case anyone needs it - convert your PG database to a SQLite database:
Also emits the PG WAL LSN so you can logically replicate to SQLite after initial import.
Listened to a number of talks and podcasts recently where the guests say that they don't need CRDTs b/c they're "too complex" but then go on to explain their custom solution which is... a CRDT ๐
Maybe this means I shouldn't mention CRDTs to some audiences?
Almost have the entire SQLite dialect covered for type generation. Preview of joins & sub-selects.
Builtin functions, conditionals, with clauses, etc. supported as well!
Next up: syntax highlighting and intellisense
Pairing with TreeSQL is going be ๐ฅ
In collaboration with SkipLabs, Riffle and
@tantaman
, we sketched out a simple SQL query extension for GraphQL-like nested query results. ๐
It's already really nice to use and will allow me to simplify my application (e.g. by removing GraphQL on top of SQLite).
Beyond excited to share that Matt Wonlaw (aka
@tantaman
) has joined
@rocicorp
as our seventh partner.
Matt is well-known in the local-first community for his work on and related projects, and I'm delighted he has chosen to join forces with us.
We've
Been pretty silent for a while. That's because useQuery(...), while a great dev experience, has scaling problems.
So I'm scaling `useQuery` style hooks to 60fps response time when working with millions of items and keeping those queries up to date as data changes.
@kentcdodds
@joeflateau
Rust has a ? operator which will automatically return an error if one is encountered. Eg: โ()?โ.
This is pretty close to exception ergonomics but typed and thus checked at compile time.
I used to think forking sqlite (e.g.,
@libsqlhq
) was crazy but I'm on board now. While SQLite is great, I just don't think it is ready for the demands of the coming years.
Just one tiny tiny slice of the mounting problems --
It's been 1 year + 2 days since got started.
My
#LocalFirst
journey started way back in 2012 with
A presentation editor that ran entirely locally in the browser.
Its remote storage solution at the time was
The obligatory TodoMVC app.
This time collaborative and replicated p2p via CRDTs in SQLite.
Still a bunch of bugs, docs & packaging to clean up but sooo close to a stable release.
The client code/interface is clean ๐งฝ
Oh my fucking god theyโre not optimistically leaving the dropped appointment in place? Just reverting and waiting for the server to send new content down?
Interesting realization that if you're syncing CRDTs through a central server you can apply whatever sort of validation logic you want on the server to preserve constraints. The validation logic just needs to write fixes as a new database version to sync back to clients.
In the refinement phase of cr-sqlite -
- decreasing crdt metadata size by 5x
- improving crr creation syntax
- adding support for resurrecting deleted items
- adding list CRDTs to the DB so you can have collaborative text w/o yjs or automerge
- improving schema migrations
Spent a week optimizing `useQuery` to get hundreds of concurrent SQLite queries to return in a single frame in the browser.
And now realizing that implementing a fully reactive SQL layer actually isn't that far out of reach.
ty &
@jmhodges
Facebook handles most of these problems by converting all queries to persisted queries at compile time. That allows you to know exactly what queries can be executed and to what depth and so on. Authz is handled at the individual node level via fwrks like
Big release today. Little did I know the cr-sqlite sync layer previously didn't work in `Chrome for Android` ๐ซ It does now!
Thanks to everyone who provided feedback and reported bugs!
@maccaw
> However I still believe that for many production use-cases you want the server as a ultimate source of truth.
๐ฏ
- Allowing servers to overrule clients
- Multi-tenancy
Are the two next big things for us to get done in cr-sqlite.
Putting together some benchmarks and examples for Materialite (that's what I'm calling this incremental view maintenance in JS library) --
I always underestimate how long it takes to get something ready for public consumption ๐คช Just a few more weeks?
Spent the am building reactive code cells that can depend on one another for the new documentation site.
Like ObservableHQ but I didn't want to include their notebook interface.
There are two ways that I love to build web apps:
Server First and Local First.
1. Server First (Remix today). HTTP+HTML are the base API, assume a connection, or talk to service workers as if you're talking to a server.
This has the potential for very fast page loads, "land
Could be a big week for TinyBase. I'm pretty excited about SQLite & CRDT synchronization in v4.0 - nearly there!
I just put up a fun TinyBase version of
@tantaman
's awesome cr-sqlite demo to give you an idea.
Let's go!
Great talk by
@aboodman
of
@rocicorp
where he introduced Zero, their new hybrid query sync engine that searches the local cache first before falling back to the server for non-local data.
Notion and Slack have so many rough edges in terms of UX.
- Code blocks that become inescapable
- Inconsistent shift+enter behavior
- Lists responding to delete incorrectly when things like mentions are in them
vs Obsidian & Discord where everything works as expected.
@captbaritone
Yeah, they're railroad diagrams and fairly common in the compiler space. The JSON spec uses them too:
and a library to generate them if you're interested --
Haven't used Synergy since high school but coming back to it, it is pretty amazing.
Can move my mouse and copy-paste between all my machines as if they were a single machine.
The bright side of landing a perf regression is taking time to add more perf tests in the post-mortem and finding a 50x perf improvement.
Pulling change sets from cr-sqlite will be 50x faster in the next release ๐
@pvh
All my intuition points to the opposite of this quote being true.
- Devices have tons of compute and storage waiting to be used
- Cloud costs are crazy
- Privacy matters to people
- Given how many devices people have, data actually lives longer when local and replicated to peers
Do any databases have direct bindings into a host language that, rather than creating SQL, creates the query program directly?
E.g., `({...}).where(...)` would not have to go to the SQL step and would go directly to a compiled statement.
On the debate between raw SQL & ORMs, I find myself flip-flopping over the years.
Is there a final answer at the end of the curve? Or just an unsatisfying "it depends"?
On my way to
@localfirstconf
. Really excited to talk about taking sync mainstream:
Thank you very much to
@schickling
for organizing and the endless energy to bootstrap this community.
@rocicorp
is proud to sponsor, and looking forward to many more
If we made houses like software: we'd have proprietary electrical sockets, our homes would only work with appliances made by the manufacturer of the home, we'd be charged a fee for exchanging money in the house.
Type-level programming melted tech twitter's brains. Not because the programs were difficult but because they look different.
Type-level programming 101 by analogy to JS constructs.
A nice property of a query language where every query is a subscription is that you can globally optimize across all queries.
Multiple queries doing the same join? Share it between all queries. One query is a more constrained version of another? Re-use the first query's output.
I find, now that my projects have traction, I'm moving at a glacial pace :(
- Messages to follow up on
- Integrations to consider
- Design input to provide
- Meetings with interested contributors
- Job offers to consider
Is muting everything and grinding a valid option?
@enlightenedcoop
lots of repliers are misunderstanding how intention is used here. Theyโre reading it as โgood intentionsโ rather than being intentional and crafting oneโs life like a work of art. Fascinating.
The biggest benefit of migrating a project from C to Rust is people actually starting to contribute. The second is not spending an inordinate amount of time reviewing code for memory issues. And lastly, being able to make large changes with confidence.
Print books have a serendipity that I often don't experience in the digital world. Stuck on a problem, opening a massive book and somehow landing on the exact page I need.
Getting tired of seeing so many query builders trying to solve the TS integration problem while introducing their own problems.
Did you know SQLite lets you encode your own type hints into create table definitions?
We can even make TypeScript aware of these w/o a query builder.
So the official WASM build for SQLite is still rather rough. Mainly due to OPFS's coarse grained file locks.
Back to wa-sqlite and indexeddb-vfs for the time being.
@kriswuollett
@jmhodges
more or less. The upside is you can ship quickly and then optimize when there's actual problems.
Of course there should still be _some_ up-front thinking in terms of expected query patterns and modeling the data in a way to serve those kinds of queries.
"Database features should be language features"
Alexander Stigsen from
@realm
shares his insights into how with local-first you want to bland the gap between the database and the language.
๐ฏ. Just make a modular monolith and let automation figure out deployment details. E.g., automatically splitting pieces that donโt talk (or barely talk) into separate deployments.
Towards Modern Development of Cloud Applications
"microservices conflate logical boundaries (how code is written) with physical boundaries (how code is deployed)."
"we propose a different programming methodology"
"Our implementation reduces application latency by up to 15x and
I'm excited about pulling in the more complicated operations () and using differential dataflow to maintain the views for an application that is entirely event sourced. Having a fully event-source app unlocks new things --
@DoomChild2
@Grady_Booch
mono vs many is an incidental detail.
You want, regardless of repo structure:
- atomic commits across _all_ your code
- atomic deployments based on a single sha
- ability to bisect the entire codebase to automatically find bugs
Monorepos are the only tool for this currently.
This is just full of gems. I've found my kindred spirit?
@CoderHi
Using SQLite as your reactive data store and single source of truth ~36 min in
Also eschewing ORMs and doing everything via store procedures โค๏ธ