SQLite takes multitenancy to the next level. Just tested
@tursodatabase
's libsql-server hosting 100k databases, querying them all in a loop. It used 1.6GiB of RAM.
I could host a database for everyone in my town on a Raspberry Pi (and still have spare memory for
@The_Pi_Hole
).
Attention everyone, we wrote a book on databases! Along with amazing
@c_a_dunlop
,
@cardeneti82118
, and Pavel, my ex-colleagues from
@ScyllaDB
. I *really* enjoyed writing the Drivers chapter. It's going to be freely available in ebook form once released as an
@Apress
Open book,
Attention everyone, we wrote a book on databases! Along with amazing
@c_a_dunlop
,
@cardeneti82118
, and Pavel, my ex-colleagues from
@ScyllaDB
. I *really* enjoyed writing the Drivers chapter. It's going to be freely available in ebook form once released as an
@Apress
Open book,
Wondering just how ergonomic
@ziglang
is? I've never touched either
@PostgreSQL
source code or Zig in my life, and yet I was able to cook a working, loadable Postgres extension in Zig in a few hours today. Amazing interoperability with C!
Details coming soon 😎
I just peeked into my wife's crocheting book... if she can make a squirrel out of this cryptic notation, pretty sure she can learn x86 assembly in a week.
Next weekend project: tursobox
Built on:
-
@orangepixunlong
Plus 2e
-
@CrucialMemory
500GB SSD over USB
-
@armbian
-
@tursodatabase
's sqld
SQL server for local dev and tests, always online and <1ms response times.
I just need to make sure sqld builds on armv7 first!
My favorite bug hunting blog by far is
@ScyllaDB
's , I reread it at least once a year.
Looking for more, please leave links to your favorites in the comments!
This is amazing, true power of open-source:
1.
@penberg
starts hacking on a Rust MVCC implementation based on the Hekaton paper
2.
@iavins
finds a bug in the paper itself, confirms it with the authors
3.
@iavins
posts a fix along with tests that prove the case
Brilliant pattern by
@TigerBeetleDB
and
@ziglang
: pass an allocator only as a parameter, so you know if a function allocates just by looking at its params. I fully recommend the whole talk!
⊕ XOR appreciation thread ⊕
XOR is such a powerful function. So simple that toddlers understand it, cheap on any CPU (1 op), can be implemented in hardware with a single logic gate, and yet it powers the modern cryptography.
Some of its magic properties are:
1. A ⊕ 0 = A
2.
You know what fits into
@tursodatabase
free tier? An entire IPv4 geolocation table. It's 280MiB, so instead of using external services, you can just put it next to your data and get everything with a local SQL query. Neat!
🏁 BEGIN EXCLUSIVE appreciation thread 🏁
SQLite transactions are lazy by default -- they only take a write lock when necessary.
If your transactions start readonly and then upgrade to writes:
```
BEGIN
SELECT ...
INSERT ...
COMMIT
```
, it may be better to just declare it a
What I find brilliant is that you get fast reads, fast writes, and only pay on the restore path. On writes, deterministic SQL statements are stored in the log, but the in-memory page cache is updated with a proper b-tree page. That makes the read path unchanged, because it reads
The crux of the paper is that they attempt to solve the write amplification problem in SQLite with logical logging. That is, instead writing out pages, they log the SQL statements after non-deterministic parts are evaluated.
C is not just a bad language. It is a demonic instrument, that leaves scars on your soul that not even time can heal.
We had a small bug recently where one extension that works with SQLite stopped working with libSQL. One of our engineers,
@iavins
, spent a whole day debugging
A significant open-source contribution from
@realwasmedge
folks just landed in
@libsqlhq
: improved memory allocation, WASI support and more efficient type translation for
#wasm
user-defined functions. Thanks!
#WebAssembly
Really nice touch by
@SpringerNature
to also publish each chapter of Database Performance at Scale as independent PDF: Only care about database drivers? No problem, here's a mini-ebook for you. Neat!
Exactly! The only acceptable way is to store a database in an image, which is perfectly doable with
#SQLite
and a little bit of steganography.
The pic of my dog attached to this tweet contains the database, go ahead and verify with `steghide`! Passphrase "dog"
Compute to data? Polyglot UDFs?
#WebAssembly
is bringing exciting new ways for developers to make use of their data without having to move data into external applications.
Learn why this is a big deal for security, performance, and database accessibility:
I already use embedded replicas at home for all kinds of weekend projects: my local
@Raspberry_Pi
server syncs with
@tursodatabase
to get fresh data, and all reads are local, so super fast, and free - you're just reading from your own disk👇
🎁Tuesday Launch (2/5)🎁
Introducing...
♾Unlimited Embedded Replicas♾
...for every Turso user.
So you can deploy anywhere, even inside your own infrastructure. Read all about it 👇
Approaches to limited memory conditions by standard libraries:
C: return NULL
C++: throw std::bad_alloc
Zig: pass your own allocator, return OutOfMemory
Rust: lol what do you mean out of memory 🤪
My libSQL talk from ScyllaDB Summit 2023 is available here:
Check out all the other presentations too, the agenda was really great, thanks
@ScyllaDB
for having me!
Spent full 15 minutes today pair programming with ChatGPT in order to create a documentation lookup tool for
#SQLite
opcodes, served right from
@fermyontech
Spin. 100% worth it, already saved me more time while hacking on new experimental
@libsqlhq
syntax!
Example:
Contrary to popular belief, the hardest code to review isn't Linux Kernel, obfuscated Lisp, or C++ metaprogramming. It's the code your CEO posted after a murderous 3 day hackathon.
🎙️Ep 20 - Database Performance at Scale with
@sarna_dev
.
Piotr is one of the co-authors of the famous book "Database Performance at Scale".
In this episode we dive deep into
- understanding performance wrt to databases
- zig vs rust
- importance of database drivers, etc.
The user-defined functions backend for
#WebAssembly
, coded entirely in
#Rust
and Wasmtime, coming soon to
#libSQL
. Here's a preview:
Next on the list - exploring integration with WasmEdge
At first I thought it weird that
@ziglang
's errors are just an enum, no payload. But maybe it makes sense... It encourages you not to use error-driven programming, where you encode program logic in throw/catch unwinding. Instead, errors are only used for communicating what
Just recorded my session for
@p99conf
. All smooth, except one embarassing misunderstanding when they suggested I smile, fully knowing I'm Polish and it's not a legal facial expression around here... Looking forward to the conference! Agenda available here:
Update: tursobox already serving queries💪
I just needed to disable Wasm support, since armv7 support for
@bytecodeallies
's wasmtime is not cooked yet. Main conclusion is that
@orangepixunlong
makes for a great local dev server for
@tursodatabase
's sqld!
Next weekend project: tursobox
Built on:
-
@orangepixunlong
Plus 2e
-
@CrucialMemory
500GB SSD over USB
-
@armbian
-
@tursodatabase
's sqld
SQL server for local dev and tests, always online and <1ms response times.
I just need to make sure sqld builds on armv7 first!
Live testing of the new lock stealing patch for
@tursodatabase
's sqld. Transactions can now execute for longer than the original 5s limit, but they will be preempted by new transactions if they passed the limit. Nice improvement!
Here's a trick I use to remember the Little's law equation - think units. Throughput is measured in 1/second, average latency is in seconds, and concurrency is just a number, so for the units to match, concurrency = throughput x latency.
Little's Law is a great tool to have in your toolbox when building for low latency!
Little's Law is a theorem within mathematical queuing theory, which we use to describe the connection between latency, throughput, and concurrency within computing systems
Just trimmed another ~50% of Wasm module size compiled from Rust by recompiling std lib with abort on panic. The demo user-defined encryption function for
#libSQL
now occupies 24KiB, used to be 44KiB.
How come I only learned about
@llvmorg
's lldb today... looks like a great alternative to gdb for quick debugging.
Loaded the symbols much more quickly, and the interface is way more intuitive, even if you're already proficient with gdb.
I highly recommend this one, it's a great deep dive into how I/O actually works (and a rare opportunity to watch Avi talk in outer space!). My personal favorite from
@P99CONF
@KunalSin9h
It sums up good C++ practices. Makes you realize variables should be const by default, std::move should be default semantics for passing values, slices give you a good perspective of why things like std::string_view make sense, etc.
Defining your SQLite table with foreign keys and ON DELETE CASCADE sounds like great idea for integrity, but it can also kill your query performance.
Say you have two tables:
```
CREATE TABLE cities(id INT PRIMARY KEY, name TEXT);
CREATE TABLE restaurants(
id INT PRIMARY KEY,
Back in the days I led
@ScyllaDB
's Rust driver, I wrote a tool for comparing our project against other popular drivers.
I checked back, and looks like ScyllaDB Rust driver became the absolute default choice, congrats to the dev team!
👇 SQLite3MultipleCiphers is super promising for giving
@tursodatabase
encryption at rest, the demo already works!
I hardcoded the encryption passphrase to "heyhey" for now, not sure if it passes all SOC2 requirements -- I need to contact my lawyer wrt. that.
One of the things I've wanted to do for a long time is enable encryption at rest in Turso for embedded replicas but also on the server side. Looking forward to seeing if
@sarna_dev
can pull of integrating into libSQL:
A
@ziglang
*contributor* ( ͡° ͜ʖ ͡°) . Jokes aside, it was really refreshing be able to just fix whatever you need right in the standard library code in `/usr/lib64/zig`, and then push the fix upstream. Distributing the standard library as code is one of the things that make Zig
While
@sarna_dev
was working on pg_turso, he at some point proclaimed that he had become a
@ziglang
programmer. Luckily for me, he's since then also agreed to write some
@rustlang
. I didn't do any of the work, but Zig worked really great for writing an PostgreSQL extension!
Huge congrats to
@PostgreSQL
for the 1st place in
@StackOverflow
developer survey 2023, but what I find most interesting is that
#SQLite
is doing so well among all the distributed systems on the list. Shows how important it is to have a local database engine around!
My takeaways from
@wasm_io
2023, day 1:
1/5 I'm glad to see that the presentations weren't browser-centric at all -
#WebAssembly
is considered to be the go-to language for virtualization, serverless, edge functions. As it very well should be!
Am I the only person suffering from microcontroller OCD? I need to regularly browse for
@arduino
,
@Raspberry_Pi
and all kinds of sensors and add-ons for them, just to eventually convince myself that I already have a full cabinet of those, some of it never used.
That said, some
Pitfall of the day: if you use mimalloc in your Rust project, beware that it runs in secure mode by default: .
Secure mode is great against memory exploits, but lots of use cases don't need that. The penalty? After ditching secure mode, resident memory
As usual,
@penberg
's post on how he's clueless about some subject created a goldmine of resources. Highly recommend the comment section of that one! 👇
I have some really basic understanding of GPU programming with shaders and CUDA, but am mostly clueless of actual GPU architecture and how code gets executed on it. What books or blog posts would you recommend me to read to catch up?
Native CRDT support will allow developers to write offline-first apps with a proper database engine - libSQL. And the fact that it's a community contribution makes it 1000x more exciting!
One of the things I'm most excited for is partnering with the LibSQL folks to create a custom syntax for defining tables which are backed by CRDTs!
LibSQL is truly getting SQLite ready for the coming decades --
Turso is available over raw HTTP for stateless communication.
If your language doesn't have a supported Turso SDK, you can still enjoy most of Turso like this.
It’s a big day for us – after over 5 years, we’re out of stealth! We’ve built a completely new approach to software reliability testing. We find bugs autonomously, with 100% reproducibility. Check out our launch blog post for way more info!
#Linux
is amazing - just upgraded my laptop disk from 1 to 2TiB as follows:
1. Connect the new disk via USB adaptor
2. dd if=/dev/nvme0n1 of=/dev/sdc conv=noerror,sync bs=4K status=progress from a live disk, the one currently used by my
#archlinux
3. Use gparted to resize the
I hit an interesting bug when using cargo lambda, based on cargo zigbuild, to create an
@awscloud
Lambda app with
@tursodatabase
's embedded replicas.
libSQL kept failing on opening the db, and some gdb sessions later I realized it failed during lstat syscall, which returned
When boss lets you expense
@openai
's ChatGPT Plus for "AI assistance on
@tursodatabase
hacking", but you're more into generating cool story books for your daughter. The Case of Missing Gooseberries:
And here it is:
We decided to write pg_turso in Zig, and it was absolutely the right choice.
But are we rewriting everything in Zig? No, not yet.
Read more and draw from
@sarna_dev
's experience!
Today
@ziglang
made me realize: people swear they never use global variables in their life (obvious antipattern), and then happily malloc() if they need memory...
Data edge in practice: I tested how long it takes for my weekend project to contact
@TursoDatabase
in 2 cases:
1. single db instance in New Jersey
2. extra local replica in Warsaw
Results: humble 32x faster response after:
$ turso db replicate mail_db waw
Details soon!
3/5 It was very nice to see
@emaxerrno
's Redpanda mentioned a couple of times. I consider
@redpandadata
one of the early adopters of
#Wasm
, and it was in fact their implementation that inspired me to add Wasm-based user-defined functions to
@ScyllaDB
(and later
@libsqlhq
ofc!)
Since when does
@github
Copilot have a chat? I just tried it on a very non-obvious, tricky
#SQLite
implementation detail, and it nailed the answer. Nice!
Given
#P99CONF
's focus on low-latency engineering strategies, it's not surprising that
#Rustlang
has been a trending topic since day 1. With over 50 tech talks over two half days, we wanted to share a sneak peek into some of the Rust-focused talks we'll be featuring.
Here are
I went a step further and implemented a
@telegram
bot for looking up
#SQLite
(and
@libsqlhq
) opcodes. It's really amazing how easy it is to hack your own bot, backed by Fermyon's
@spinframework
. Each time you send a message, Telegram server sends a request to my Spin app, all
Spent full 15 minutes today pair programming with ChatGPT in order to create a documentation lookup tool for
#SQLite
opcodes, served right from
@fermyontech
Spin. 100% worth it, already saved me more time while hacking on new experimental
@libsqlhq
syntax!
Example:
TOMORROW -
#ScyllaDB
is hosting a [free + virtual] "Database Performance at Scale masterclass" on Wednesday, taught by authors of the book. We'll cover the impact of workload characteristics, database internals, database driver optimizations & more.
Here's why pushing data to the edge makes so much sense — without middlemen, you can get your data in less than 30ms total, especially if those middlemen happen to be on another continent. (While it may be shocking to some, not everyone lives in close proximity to us-east-2 😇).
Get your data in 28ms - that's what Data
@Edge
means! Inspired by an amazing
@vercel
latency playground by
@leeerob
, I decided to add an option to test the latency of the browser calling straight to the DB. For my location (Warsaw), I can get data from
@tursodatabase
in 28ms :).🧵
Today I learned that NVMe PCIe to USB adaptors exist and cost $10. And that means my old laptop SSD will gain a second life as a very compact (barely 3.5'') external 1TiB drive.
I wish I realized sooner that KubeCon was going to co-host CloudNativeWasm day... I'll save the talk submission for next year, and meanwhile I recommend everyone to attend, esp. that
@fermyontech
supports the event -- these guys tend to make conferences awesome (ref:
@wasm_io
)
🎙️Ep. 15 - Understanding
@tursodatabase
with
@penberg
🚀💡
It was such an insightful discussion for me -
- Cpu-centric POSIX
- PhD and Startups
- Unravelling TursoDB - why embedded database? The HA choice for Turso, etc.
#TheDistributedFabricPod
Need a temporary email address?
@sarna_dev
explains how he built a service from scratch with
#Turso
and
@rustlang
, free to use, and with full source code available. Start reading here:
I'm really proud
@tursodatabase
is part of this. Cloudflare folks provide so many innovations that one can't really imagine the Internet without them anymore 🤷
Earlier this year, we introduced integrations with Supabase, PlanetScale, Neon and Upstash. Today, we are thrilled to introduce our newest additions to Cloudflare’s Integrations Marketplace – Sentry, Turso and Momento.
#BirthdayWeek
That's going to be my first onsite talk in a looong while, looking forward to it!
Spoiler alert: maybe I'll also slip in a bonus slide or two on how to connect to libSQL from native Wasm environments like
@fermyontech
's spin and
@cloudflaretech
's Workers😇
Learn about
@libsqlhq
, a fork of SQLite, and its support for dynamic function creation and execution directly via SQL, all powered by Wasm in
@sarna_dev
's talk "WebAssembly functions for your SQLite-compatible database"
#wasmio23
When should you pick SQLite as your production database?
Although many people are (re-)discovering SQLite, there are still a lot of popular myths about it, but perhaps also some hype that fails to mention the downsides.
First and foremost, SQLite is a versatile,
Today, we are adding encryption-at-rest natively to libSQL. It is fully OSS, and all you have to is to add a key parameter to the constructor.
This works for local files, and you don't even have to be a Turso user for that.
SQLite doesn't have encryption at rest as an Open
As a heavy user of , I decided to start working on a clone, powered by
@ChiselStrike
Turso and
@libsqlhq
. Half-baked, but works! All e-mails sent to the @ domain will be printed on the webpage:
That's going to unlock very interesting use-cases for
@libsqlhq
as well - we already have all the code merged from upstream SQLite. Can't wait to give `./configure --with-wasi-sdk=$WASI_SDK_PATH` a try!
We are happy to share that
#SQLite
has out of the box support for wasm32-wasi since 3.41.0.
Here you have the story behind those changes and how we collaborated with the SQLite team 🤩
#webassembly
Damn, this is an excellent resource.
I read the first few chapters closely and plan to return to the rest. Lots of interesting stuff + frank discussion of how to truly evaluate different database options.
Also, helpful hints for sounding smart 🤓. Very good tone throughout!
4/5 Now that
@libsqlhq
user-defined functions can run both on Wasmtime and
@wasmedge
, I wonder if
@ScyllaDB
wouldn't benefit from a second runtime as well, esp. if the "reactor stalls" issue could be solved with tools like asyncify (). Food for thought ;)
One weekend project I actually use is disposable email: . What I really like about it -- any username works, so you can sign up on any webpage without generating the inbox first, and later just check the email. Nice alternative to !
2/5 I really enjoyed all the off-stage conversations, it's nice to see that both speakers and attendees are really serious about
#Wasm
, you can learn a lot just by being there. I'm definitely up for WasmIO 2024.
I really need to get back to , I already got stuck 2 times at Challenge Set 3, this is an immensely educational page. (and it's crypto as in cryptography, the cool one)
@glcst
when joining
@ScyllaDB
, my first thought after learning about the company history was "I wish I found these guys before they pivoted from
@OSv_unikernel
, sounds like the coolest project ever"
@LambrosPetrou
@tursodatabase
@The_Pi_Hole
The scenario tested here is "database per user", so there's no common schema to coordinate - each user can store whatever they wish in the db