Every time I've encountered a database without foreign key constraints and let the "app layer handle it" I've found data inconsistencies, every single time.
What you should do to preemptively help future performance on a new Postgres database? These tips will save the day for you when you need to investigate performance 🧵
Talking with some folks about worry on scaling and how far Postgres can really go... benchmarked our $35 a month database plan pushing 5,000 transactions per second.
That's 432,000,000 per day. When you see numbers X million or Y billion transactions, do the math.
$35/mo 432m
This is so simple yet brilliant, if you want JSON/documents in a particular form here's a great post on how to let Postgres do the heavy lifting of returning it for you -
SQL has been one of the most valuable skills I’ve picked up in my career, here are a few thoughts on why the return on investment of learning it is well worthwhile:
If you think SQL can't be readable... you probably haven't used CTEs (common table expressions) or "WITH" clauses to write SQL with composable blocks.
CTEs and proper indentation and you SQL can start to look more like code and way less scary.
Way back at Heroku we had this idea of team swaps/internships. You could swap teams for a short period of time to get a sense of their world of engineering. For a lot of engineering teams with similar toolsets this worked out pretty well.
Is Heroku in a good spot? No.
If you can't appreciate what Heroku did for developer experience then you shouldn't be talking how you're creating a great developer experience.
Respect your elders.
Excited to have joined
@crunchydata
to help build our their managed service offering, apparently the only thing I know how to do is run Postgres for people, at least it's Postgres though
I love the metrics of people "my app is scaling amazing because of shiny new serverless tech"... when the metrics are like 10k transactions in a day...
And I'm just over here watching Postgres (a single server) push several 100k transactions per second.
Final day at Microsoft as part of the Azure Postgres team. I'd say lets catch up but well getting out not so much an option these days. Drop me a note would love to say hi virtually.
Ready to unplug for a while for a good break and build quite a few lego sets.
ORMs aren't bad, but they do get over-used at times.
Great for: Simple updates, CRUD operations, very simple aggregations, managing your migrations
Mixed results on: Joins, subqueries
Bad results: Generating reports
Don't be afraid of jumping to SQL *when you need it*.
I work with Postgres daily and sometimes it still takes a bit for me to parse the output of an explain plan, this walkthrough is so handy if you're not a full time DBA in understanding a bit more about your Postgres query performance
Really excited to announce Crunchy Bridge - a modern Postgres as a service from
@crunchydata
There is a lot already in here and a lot more to come. Excited to make Postgres available and even more awesome for devs
Conversation with a VC last night...
VC: Oh, you know some about Postgres
Me: Yep
VC: I was at LinkedIn way back, we bet on MySQL
Me: Not crazy for that time
VC: Postgres really came out of nowhere in last few years, any thoughts on how that happened?
Me: How much time you got?
If I were a VC or PE I'd be pulling together some money to see if I could buy Heroku off Salesforce.
It still solves a problem that nothing else has come close to at that scale.
(Lots of earlier smaller players, but if you want to place a big bet it would be a good one)
The number one cause of production database outages I see these day is because of CDC tools. Generally the lifecycle of a tech company goes something like:
We start building an app and choose Postgres (duh.) Fast forward a little and "we need to get insights to the business".
Said it before, but (cause people) feel like I need to say it again... the make money stack:
Rails/Django and Postgres
Throw in tailwind for design for good measure
It just works. You can build a really good business that way.
Did you know you can name/save queries in your psqlrc that are common database tasks? Inside your psqlrc:
\set long_running 'SELECT pid, now() - pg_stat_activity.xact_start AS duration, query, state FROM pg_stat_activity WHERE (now() - pg_stat_activity.xact_start) > interval ''5
pg_stat_statements - easily most useful Postgres extension that exists. Without becoming an expert in Postgres you can still get a sense of how queries perform and what to optimize -
Repeat after me: New shiny tech/framework does not magically solve all the problems from yesterday, it is more likely a layer of abstraction and complexity. Boring tech is great for building real businesses.
IOPS are a thing that baffle a ton of folks, providing massive IOPS out of the box for all Bridge customers has been one of the single biggest performance wins our customers see in migrating to us. Here we breakdown IOPS and what it means for your DB -
What you should do to preemptively help future performance on a new Postgres database? These tips will save the day for you when you need to investigate performance, spend 5 minutes now and get your database production ready.
1. Install pg_stat_statements, no matter what it'll
Just heard from a customer that migrated to us from Aurora because they were having connection management and scaling issues on it.
What solved their problems? Vanilla Postgres.
You probably don't need Amazon or Google scale, you just need Postgres.
Few Postgres psql config recommendations (set in your .psqlrc in your home directory):
\set QUIET 1
\pset null '👻'
\x auto
\timing
\set HISTFILE ~/.psql_history- :DBNAME
\set HISTCONTROL ignoredups
\set COMP_KEYWORD_CASE upper
\unset QUIET
Tech is a very small world, treat people well, do good work.
The game is a lot longer than you think and you'll reap benefits for doing the right thing or pay the price for taking shortcuts.
It is well known I’m a psql guy personally, but super excited to see
@Microsoft
bringing some of their expertise of graphical editors to the
@PostgreSQL
community -
If you're on Heroku:
Rotate all config var creds.
Change password.
Rotate API Key and 2FA.
Audit as much as you can of Heroku releases and deploys.
It's not gonna be fun, but do it.
Have a cousin in college that took a DB course, they just shared their slides and lab exercises with me. No wonder devs hate databases and SQL, we can do so much better on educational content.
Anyone have favorites for learning SQL/DBs?
Postgres for analytics has always been a huge question mark. By using PostgreSQL's extension APIs, integrating DuckDB as a query engine for state-of-the-art analytics performance without forking either project could Postgres be the analytics database too?
Bringing an analytical
Feels like we're breaking every rule of "don't do this with Postgres"
@crunchydata
and it's working out just fine.
Using it for analytics/metrics✅
Storing logs in it ✅
Launching docker containers from inside it ✅
Running with pl/python ✅
Ingesting 50m events per day, on a $190 a month database, providing dashboards and reports ranging in response time from 1-2ms up to 100ms all depending on the range and amount of data reported back, here's how. Along with future optimizations -
Pro-tip
#PostgreSQL
tip, trying to use psql more, but still missing a richer editor when writing SQL? Set your $EDITOR or $PSQL_EDITOR to the editor of your choice. Then use \e to open the last executed SQL in that editor, close and save, it'll execute the query.
To recap:
✅ pg_stat_statments
✅ log queries over 100ms
✅ log explain plan of queries over 100ms
✅ auto-kill queries over 30s
And your Postgres will be 💯
Happy Postgres 14 release day ().
Excited to already have it officially supported on Crunchy Bridge ().
If you want to get your hands dirty with PG 14 feel free to create and account and give it a spin.
For as powerful as it is, SQL isn't exactly a user friendly language to work in. Debugging SQL can be slow and painful. But, you can write more legible SQL by taking advantage of one of the least used functionalities: CTEs (Common Table Expressions).
CTEs are essentially a
Working on something fun for
@crunchydata
website update that I think will be one of the most useful Postgres resources in a single place over past 10 years.
Few weeks off, but this should be fun to get out the door. 🔥
1/ In every startup I've ever encountered the most challenging thing is focus. The first step is to understand what you're building, what your model is, and then how you sell it. Once you do that perfect it.
I frequently get on calls with folks wondering how their Postgres database is doing, this is the playbook I run through in 10-15 minutes for how healthy things are and finding low hanging fruit -
My tweet on what could be improved in Postgres got way more response than I anticipated. While I agree with a ton of it, feels worth laying out Postgres is awesome, so here goes:
One of the biggest features that we were wrong about at Heroku Postgres was dataclips. It was the vision of a single person that pushed it through against all the collective teams opinion.
Beyond excited to announce the release of Crunchy Bridge for Analytics–easily working with your data lake. You get the Crunchy Bridge experience plus
* Seamless S3 integration
* Easily query CSV/Parquet data
* Vectorized query execution
All in Postgres -
I have failed as a parent...
My daughter just now: Jedi's don't have powers they just use lightsabers
I'm taking tomorrow off and my kid is gonna be out sick, we're starting at episode 4 at 7am...
I bet most folks didn't even know there has been an entire machine learning extension () for Postgres for 10 years now...
Why? Because none of the providers of Postgres have supported it...
Changing that starting today -
Back at Heroku when we built dataclips most of the Heroku Postgres team was against it and thought it was a terrible idea. One person on the team pushed it through. The rest of the team was wrong, it turned out to be brilliant, like github gist for your data–against a live
Thinking about my angel investment thesis... tempted to go with I invest in the make money stack "Rails/Django, Postgres/MySQL, Heroku, GitHub"
If you're building on this you've just increased your chances of success 10-fold
Lots of talk with customers lately about multi-tenant database design patterns, took a few minutes to highlight the approaches with a special guest star of Citus -
Lots of backchannel talk on
@heroku
.
Many of us don't want to speak out when things happen, incidents happen, it sucks. But let them manage it and root for them.
But a lot of us feeling strong sense of responsibility to what we once built because the comms/response not okay.
It's at least once a week I talk with someone that "loves Postgres" but isn't sure why... which evolves into me ranting for 2 minutes on all the awesome features.
In hopes sharing a little broadly here it goes 🧵
Having a lot of conversations with folks lately "can Postgres be used for metrics/analytics?" Short answer is yes. Some deeper blog posts coming, but we dogfood our own stuff
@crunchydata
and just checked... we're handling 50m events daily currently for Crunchy Bridge.