JP, the creator of FizzBee, talks about formal methods, TLA+, distributed systems verification, and a better way forward.

Chris Riccomini talks about the current state-of-the-art in data pipelines and data warehousing, and shares some of the solutions to current problems dealing with data streaming and warehousing.

A cloud native embedded storage engine built on object storage. - slatedb/slatedb

Data is at the center of many challenges in system design today. Difficult issues such as scalability, consistency, reliability, efficiency, and maintainability need to be resolved. In addition,...

Work with your web service, database, and streaming schemas in a single format. - gabledata/recap

Be wary of schemaless data structures, since they still have an implicit schema and, with a couple of exceptions, an explicit schema is better.

Materialized View Capital invests in early stage infrastructure startups.

Register for Kafka Summit: https://kafkasummit.ioLet's begin a very unusual Kafka Summit by reflecting about change. Changes we've seen in the software engin...

We're excited to announce the public beta of Bufstream, a drop-in replacement for Apache Kafka that's 10x less expensive to operate and brings Protobuf-first data governance to the rest of us.

Python SQL Parser and Transpiler. Contribute to tobymao/sqlglot development by creating an account on GitHub.

Open Robo-Advisor is a flexible robo-advisor library written in Python. 🤖 - highwire-ai/open-robo-advisor

Flexible deployment is now table stakes. Infrastructure must run embedded, client-side, single-node, clustered, as SaaS, BYOC, and self-hosted.

Why are engineers taking databases apart and putting them back together, again?

Introducing Materialized View, a newsletter offering software infrastructure news, developer interviews, stack deep dives, and project and paper highlights.

And Why Your Data Pipelines Don't Scale

An infrastructure engineer, focused on distributed storage system

We open sourced SlateDB a week ago. Let's look at where we're at and where we're headed.

Apache Kafka is an aging open source project. It's time to accept that Kafka's protocol is what matters.

The Missing README gives new engineers a masterclass in coding practices, technical skills, and tips for workplace success.

0:00 Talk30:13 Q&A

Why Samza failed, how it led to Kafka Streams and Kafka Connect, and why I'm skeptical of Apache Flink.

Modern columnar data format for ML and LLMs implemented in Rust. Convert from parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, Du...

Realtime OLAP is colliding with search, observability, and data warehouses.

A look at the excitement around ClickHouse. I break down what makes it great, and look at the challenges ahead.

Big data is dead. Long live easy data.

A brief history of Avatara, Apache Pinot, and Apache Druid.

It's silly to have applications generate text-based SQL; they should be allowed to pass query plans to the database.

Everything you need to know about the new AI/ML storage formats from Meta and LanceDB.

Chatting by chance

With an effort to make Notion faster and more reliable for years to come — we migrated Notion’s PostgreSQL monolith into a horizontally-partitioned database fleet.

FastAPI framework, high performance, easy to learn, fast to code, ready for production

the portable Python dataframe library

I'm squarely in the trough of disillusionment with S3.

Software infrastructure hot takes, projects, papers, developer interviews, and deep dives. Brought to you by Chris Riccomini. Click to read Materialized View, by Chris Riccomini, a Substack publica...

There’s been a surge in durable execution frameworks over the past 6 to 12 months. Temporal has been the go-to for a while and now challengers are emerging.

Discover more about what's new at AWS with Amazon S3 now supports conditional writes

The hardest problem in durable execution, as in many areas of infrastructure, is safe updates.

Distributed object store. Contribute to linkedin/ambry development by creating an account on GitHub.

A new theorem for primary persistence on object stores.

Compare tables within or across databases. Contribute to datafold/data-diff development by creating an account on GitHub.

Databricks and Snowflake are talking a big game, but they've given us empty Github repositories and partial implementations.

Philippe Noël is CEO and co-founder of ParadeDB. In this post, Philippe and I discuss ParadeDB, the experience of building as a PostgreSQL extension, pg_duckdb, pg_lakehouse, and more ...

Netflix operates in 190 countries worldwide and we have 125M+ customers and growing. At any moment, around the globe, millions of customers are getting charg...

Microservices aren't the only way to get high cohesion and low coupling.